OpsMgr: Why all my agents turn gray - OpsMgr Healthservice must run under local system account!!!
The other day, I was working with Quest support for the eXc Blackberry monitoring software. We tried to delete/import their MP. During the process, I will need to input the password for the account which OpsMgr SDK Service and OpsMgr Config Service are running under. I found the password and I want to verify it. I didn't use the SDK service to verify concerning that if I don't have the correct password, I will not be able to restart the service. So I use OpsMgr Healthservice instead thinking I could easily switch back to local system account. I did exactly that and a few minutes if not seconds later, I noticed that all my agents are grayed out (unknown status!).
At first, we thought this is because we deleted/import the MP -bogs down SQL database. I thought to let it run over the weekend and hoping that it will fix itself. On Monday, all the agents are still grayed out only the agent for the RMS is green. I called PSS support, MS engineer determine this is because "Duplicate SPN". He did Setspn -l which didn't catch it or didn't catch all the duplicates. He then use MS internal VBscript to identify more duplicates. The moment after we clean up the "Duplicate SPN", we restarted the RMS and the agents are starting coming back green!
I mentioned that the OpsMgr Healthservice account "swap" to MS engineer but he didn't link this two events together. I did some digging and found out that's exact the cause why all my agents turn gray. The Healthservice account (especially on RMS) must run under local system account or you will get duplicate SPN. Even after you switch the account back. Before I thought running under local system account is preferred but not knowing it is a "MUST"! I never realized that simply by change the OpsMgr Healthservice account on RMS can cause so much trouble and it costed me a support ticket. :)
Next time if all your agents are grayed out - Check your SPN first!