We were finally able to configure an environment where we could debug the issue as it occurred. This allowed us to confirm the root cause of the issue. Here is a summary of the issue and the status of the fix:
SYMPTOM:
The Software Update Point (SUP) stops synchronizing unexpectedly when Windows Server Update Services server (WSUS) is installed onto the same machine as the System Center Configuration Manager 2007 (SCCM) site server.
Upon further investigation, WSUS is found to no longer be fully installed such that the WSUS virtual directories are missing within IIS and the Update Services service is missing from the Services console.
The SCCM WSUSCTRL.log (drive:\Program Files\Microsoft Configuration Manager\Logs) may report errors similar to the following once the site server is in-state (i.e. WSUS is already uninstalled):
"The installed WSUS build (0.0.0.0) does not have the valid and supported WSUS Administration DLL assembly version. Please install WSUS 3.0 (minimum 3.0.6000.367) or above"
Status Message ID’s 7003 and 7004 are also generated at least once an hour until WSUS is reinstalled.
CAUSE:
This behavior has a two part cause:
The first part is the root cause of the issue and is due to a race condition in our Microsoft .NET Common Language Runtime (CLR) Fusion initialization code. This .NET CLR Fusion race condition issue becomes exposed by two of our System Center Configuration Manager components, WSUSCTRL and WCM, when these two components' threads simultaneously execute a health check on WSUS during startup. As a result of this issue, WSUS appears to be missing some of its binaries so a Windows Installer repair is invoked which exposes a WSUS installation bug where WSUS becomes partially uninstalled which is the second part of the cause.
The sequence of events is as follows:
1 - The wsusctrl thread and the WCM thread enter the Fusion code simultaneously.
2 - One of these threads begins Fusion initialization in mscorwks.dll
3 - Due to the race condition, the other thread proceeds with creating the assembly cache while the other thread is still initializing Fusion
4 - Because Fusion is not initialized properly, the thread that proceeded will utilize GAC paths that are not properly initialized (i.e. C:\Assembly)
5 - Due to the improper initialization, WSUS assemblies will not be found
6 - CLR calls WSUS installer to repair the installation
7 - Due to a known problem with the WSUS installer's repair functionality that has been addressed in WSUS 3.0 SP2, WSUS repair fails and WSUS is left uninstalled.
TENTATIVE WORKAROUND:
The current workarounds to mitigate the issue until the patch is released include:
1. Monitor the server for an occurrence and simply remove WSUS while choosing to keep the current database and content and reinstall WSUS using the existing database and files.
2. Remove WSUS from the SCCM site server and install WSUS onto a separate machine.
RESOLUTION:
We have a fix that has been scheduled to be included in SCCM 2007 Service Pack 2. Our tentative release date for SP2 is October 2009.
Since we have no further actions to complete with you at this time, I will close the support case while we await the release of the fix within Service Pack 2 of SCCM 2007. If you have any questions or concerns in the interim, you may contact me directly regarding this support issue.