Lessons Learned #1 - Renewing and Troubleshooting Certificates/Enrollment Issues
In the spirit of sharing, some recent experiences while troubleshooting a MDM RTM installation that may be interest for others..
Problem Scenario: Working MDM RTM instance installed 6+ months ago. Existing enrolled device where working fine and bringing up VPN. OTA Enrollment was no longer working. Error displayed on devices "Unable to enroll the device in the company domain. Make sure that the device has the correct date and time and then try again."
Troubleshooting: Triple checked all servers and devices tested. Date/time all correct! No event log entries with a smoking gun. Did notice that the Enrollment IIS log barely had any entries. Verified that the Enrollment site published through ISA was correctly configured. The Enrollment ShouldEnroll test came back with a successful “0” response..
Solution: Finally noticed that the main certificates and intermediate CA were all expiring a week from now, but still valid. Turned on ISA monitoring during an OTA enrollment and saw that the web connection was dropped very fast and explained the lack of data in the IIS logs. I checked the ISA listener rule for the enrollment web site and found that that cert was expired over a week ago and the root cause!
Resolution steps:
Seeing that all the certificates and intermediate CA server needed to be renewed anyhow, figured I would be daring and forego using Resource Kit cert tool and renew everything manually like a newbie might try.. :-)
All the certificates on the domain joined DM/Enrollment server were fairly easy. In IIS bring up the properties on each web site -> Directory Security -> Server Certificate -> Renew -> Online.
Even the intermediate CA wasn’t too bad, with it’s Action-> Renew Cert once I had the right session permissions. The ISA Listener and Gateway Web cert I did manually through the http://ca-server/certsrv website.
However to my horror and shock now newly successfully enrolled devices where not bring up VPN. The MDM VPNDiag tool showed Cert Chain errors on the device with “NAME_MATCH” errors. Double checking the certificates once again (did I mention I now hate certs?) it dawned on me that since the Gateway server is not a domain joined machine the new intermediate CA cert was not there and had to be manually imported. But even after doing so, the newly enrollment devices still had a chain error.. Also on the Gateway server MDM Mobile VPN Connections Event Log I saw these entries:
Once I rebooted both the Gateway and DM servers and enrolled another device from scratch, it finally all worked.. This leads me to believe that either the Pre-enrollment or the DM services or the Gateway services had some memory of the previous intermediate CA cert (still valid for another week) and got mixed-up negotiating the VPN connection..
Lessons Learned:
- Date/time device enrollment errors could be issues with accessing the enrollment site and check it’s full access path, proxy cert etc.
- Always use MDM VPNDiag tool from the Resource Kit Client Tools on the device to troubleshoot and verify the cert chain!
- Perhaps best practice to reboot the DM and Gateway servers after changing an intermediate CA or root CA cert. :-)
- Recommend looking at using the MDM Certificate Tool from the Resource Kit Server Tools to simplify the renewal process.
|\\arco..