So I have previously blogged about the issue of the ADFS server and the Microsoft Online Services needing to have the shared certificate regenerated each year, http://office365evangelist.com/?p=489. So I would like to expand on this issue as it affected me directly last night. Not in the way most would think. Around 2:30am my mobile phone starting blowing up on my nightstand. Side note, I am now a Consultant, I really don’t give out my home number and my mobile phone, a very beautiful Nokia Lumia 900 Windows Phone, turns into an alarm clock and it goes silent. So it was not until but wife noticed the lights flashing from over 12 calls that she nudged me to check my phone.
I also noticed several text messages and a couple voice messages that gave me a clue to the culprit. I called the last number back, it was neither a local number nor a number I had in my contact list, but knew who I was calling. It was a great client I had at my previous company. It was the top guy, the IT manager of my first ever Office 365 implementation, just about a year ago while Office 365 was still in Beta, that just now should raise a flag for any of you who read the above linked post. So what they were facing is the same issue Catapult faced a couple months ago. Their ADFS internally shared certificate with Office 365 had expired! While this is a major issue for a company that has implemented ADFS for SSO with the internal Active Directory should not be a major issue overall, it was. The reason that it was a major issue for my friends is that after getting ADFS up and running they utilized their federated identities to administer Office 365 and kinda forgot about the original Cloud Based Admin account or any other Online Only Accounts. Thus when ADFS failed to authenticate the end users for access to mailboxes, Lync and SharePoint Online, they were also unable to authenticate for Administrative access.
So the reason while I was selected to get woken up was not to catch up on the old days, share stories and have some fun; it was to beg and plead to get the Online Admin accounts username and password. Unfortunately I was pretty groggy at 3am and could not think straight. It was suggested that my SMCNEILL Online account was still alive, so I went through my standard passwords for my clients. This did not work, but because I use standard passwords for accounts it will benefit us later in this story, stay tuned. While I was working with the top dog on his cell phone, I could hear in the background another one of the admins arguing with Microsoft support. The main issue my friends had was not that they didn’t know the password for the Online Admin account, let’s just call it admin.company.onmicrosoft.com; the issue was that in the Beta days and the first six or so months of Office 365 Microsoft did not anticipate needing to reset the password for the Online admin accounts. Today if you sign up for Office 365 and you login with the company.onmicrosft.com account for the first time you are presented with a pop-up that asks not only for a secondary email address, and this cannot be for an address within your tenant domain list, but also a mobile (smart) phone number to be able to text and email a password change that is outside of the Office 365 environment. This was a great move in the last six months by Microsoft to recognize the need for an alternate way to reset a password for an Admin account. Back to my story, my friends had not logged into the Online Only account in probably nine months. So without setting an alternate email for a password reset associated with the Admin Online Account, Microsoft was having trouble validating and ensuring that if they reset this password and gave it to these yahoos on the phone what was the security risk (I’ll talk about this particular client below). But when you think about it, you have to admire Microsoft for being hesitant to reset a Global Admin User password to just someone who calls in and complains.
After trying several of my standard passwords for the SMCNEILL@company.onmicrosoft.com account and having no luck it sounded like Microsoft was close to being satisfied with my friends various verifications methods to getting the true admin account password reset. So I went back to sleep, or at least back to bed. So if you are like me, I can’t stand leaving an issue open, so my mind was racing and could not get back to sleep. After about 20-30 minutes I ended up getting up again and booting up my new company laptop. Thankfully I started using OneNote for all my projects with my previous company about two years ago and keep the usernames, not password, in OneNote pages for my clients. Well I Opened the one note for this client and noticed another thing I did not realize while sleep deprived, I did not setup my Admin account as SMCNEILL@company.onmicrosoft.com by it was Consultant@Company.onmicrosoft.com. Again in my OneNote page the password just said “Standard Client Password”. So I knew that the passwords I tried with the SMCNEILL account would most likely work with the proper user account. SUCCESS! I was able to gain access to my previous client’s tenant, I did have to reset this password since it had probably been nine months since this account was used but I had access. I immediately called my good buddy back and told him I was able to login to the tenant. He was able to reset the primary Admin account (since my account still had full admin rights) and get the ADFS shared certificate re-generated between on-prem and Office 365 and the issue was resolved.
So what is the point of this little story? Well if you have ADFS and SSO setup, DO YOU KNOW THE ONLINE ONLY ACCOUNT NAME AND PASSWORD? I agree that my friends probably failed a little bit in not having the username and password documented. But I want to use this real world example, as a warning to all of the Office 365 Clients out there! Here is what I suggest an Office 365 client does at a minimum every Quarter (3 months) but recommend every month, as part of a disaster recovery plan:
- Ensure you have the Online Admin Account Username and Password documented
-
Ensure that you have set an alternate email address for the Online Admin Account
- If you have not logged in with the Online Admin Account recently, do it!
-
If it has been awhile logging in with the Online Admin account you will be presented with the requirements to set a phone and alternate email for password resets
- If you are not sure what is the alternate email address is you can find it and reset from the properties of User Account properties
- If this is set to you companies email address hosted on Office 365, change it! To an email address outside of Office 365
- If you don’t know the account or password, utilize your Federated account to login and locate and reset the password and then login with the account (see above)
- Create a plan to document your Online Only Account username and password
- Test a login every month with your Online Only Account
- Depending on the size of your company, create a secondary Online Account with Organization Admin permissions and verify monthly or quarterly
I strongly recommend that any company on Office 365, especially with ADFS and SSO implemented, they follow the above steps. My great friends and client would also recommend this I think. You need to ensure your “Backdoor” is still a viable option to be able to access your tenant in the event anything happens to ADFS.
Side Note:
I do appreciate that Microsoft was very careful with resetting an admin account for an Office 365 Tenant! But I think in this situation they could have done just a bit of research as the licenses for the tenant were tied to a Enterprise Agreement (EA) and the worst they could have done was send the password reset info to the contact of the EA. While this will not work for every client, in this situation my friends were on the phone from 4pm their local time, and then finally decided to call me, conveniently while I was in deep sleep, but I was glad I was able to assist.
