All things SMS, System Center Configuration Manager, Active Directory, Group Policy, Virtualization, Security, Gadgets, Technology, and the Daily Thoughts of an SMS Engineer named Anthony Clendenen.

The Daily Ramblings of an SMS Engineer

Why Patch Tuesday Turns Into Patch Thursday or Friday or Monday

October 09, 2007

I used to hate the second Tuesday of every month.  I honestly liked it better when patches were released as they were ready.  What happens with the scheduled updates is that everyone knows that it is coming and that means that everyone expects their SMS Engineers to have the patches deployed that night, no matter how late, no matter how many people, no matter what!  And on Wednesday morning, usually around nine, everyone wants to know the status of the deployments.  For me I had some advantages, one I controlled the entire SMS hierarchy, everything from the design, the site servers, the physical servers, collections, packages, ads, reports, you name it, I was also on the west coast, when Microsoft didn't release the catalog until noon, it was noon, not three in the afternoon or 11 PM.  I also didn't have other managers trying to tell me what to do, occasionally I would get a call from the IT director our someone higher up but when I uttered those three little words, "It's patch Tuesday", 99% of the time that conversation was over.  So I could close my door, not answer my bat phone, turn off my work cell phone, and make full use of my time. 

But here's the thing, my situation was unique, most SMS Engineers either have a ton of other responsibilities, a ton of competing requests from people, don't have that kind of flexibility in their scheduling, don't have that kind of control over the SMS hierarchy, don't live in the PST time zone, or any combination of these including but not limited to all of these!

And if you are a manager and are reading this you may think to yourself, "they just need to manage their time better" or "these are just minor roadblocks that they can over come" or even "it's their job and if they can't do it then we will find someone else who can," I have a couple questions for you.  First, have you ever read a KB article?  No, no, no, not this summary...if that is what you read, and even if you read the entire KB article did you miss that one little piece about known issues, you had to click on a link, go to another page, and then click on another link, and then scroll down where you were presented with a little more information on what you may encounter, it reads like so...

Known issues

• During the last two steps of the SharePoint Products and Technologies Configuration Wizard, you may receive the following error message:

Failed to start service SPSearchServiceInstance on this server after completing upgrade. Please start it manually.

However, the SPSearchServiceInstance service was actually started after the SharePoint Products and Technologies Configuration Wizard finished. You can safely ignore this error message and the error message that is logged in the SharePoint Products and Technologies Configuration Wizard log file.

• The Microsoft GroupBoard Workspace 2007 add-in template for Windows SharePoint Services 3.0 may cause the SharePoint Products and Technologies Configuration Wizard to fail during a build-to-build upgrade.
For more information, click the following article number to view the article in the Microsoft Knowledge Base:

941678 (http://support.microsoft.com/kb/941678/) SharePoint Products and Technologies Configuration Wizard does not finish successfully on a computer that also has GroupBoard Workspace 2007 installed

• When you run a gradual upgrade of a Windows SharePoint Services 2.0 site collection to a Windows SharePoint Services 3.0 site collection, the server may be brought down. You must allow the gradual upgrade to complete before you apply this security update.

• You use quiet mode to install Windows SharePoint Services 3.0. If this security update is applied during the installation, the SharePoint Products and Technologies Configuration Wizard unexpectedly runs after the installer program has finished.
To resolve this issue, extract this security update to a folder on the computer. Then, copy the files to the Updates folder of the program's release version. After the extracted files are copied to that location, the folder is ready to be used to install the release version of the program that is updated to this security update level.

• After you install this update on a computer that is running Microsoft Windows Small Business Server 2003, the Web sites may not restart within the time-out period. Therefore, you may be unable to access the Backup, CompanyWeb, Microsoft Server ActiveSync, Monitoring, Outlook Web Access, or Remote Web Workplace Web sites. To resolve this problem, use one of the following methods:

• Open Internet Information Services (IIS) Manager, and start any Web sites that are stopped.

• Restart the computer.

• If you run a virus scanning program during the installation of this security update, you may experience intermittent issues during the installation. To resolve these issues, turn off the virus scanning program before you apply this security update.

• You use host-named site collections in Windows SharePoint Services 3.0. If you have many host-named site collections in your deployment, you may experience severe performance issues while this security update is being applied. For example, this issue may occur when there are more than 50 host-named site collections in your deployment.

Microsoft is aware of this issue in which the installation and upgrade will take a long time to update the databases when you run the SharePoint Products and Technologies Configuration Wizard. If you have many host-named site collections in your deployment, we recommend that you do not apply this security update at this point.
Note If you do not use host-named site collections in Windows SharePoint Services 3.0, you can safely apply this security update.

• Consider the following scenario. You unprovision a Windows SharePoint Services 3.0 search service database by using one of the following methods:

Method 1
You run the following command-line:

%ProgramFiles%\Common Files\Microsoft Shared\web server extensions\12\bin\STSADM.EXE" -O SPSEARCH -ACTION STOP -F

Method 2
You use the Central Administration page to stop the "Windows SharePoint Help Search" service. To do this, follow these steps:

1. Click Start, point to Administrative Tools, and then click SharePoint 3.0 Central Administration.

2. Click Operations, and then click Services on Server under Topology and Services.

3. Click Stop to stop the Windows SharePoint Help Search service.

You perform a basic installation of Windows SharePoint Services 3.0 that contains the unprovisioned search service database and this security update. When you run the SharePoint Products and Technologies Configuration Wizard in this scenario, the wizard is not completed successfully.
To determine whether you are experiencing this issue, open the latest PSConfig log that is saved in the following location:

%ProgramFiles%\Common Files\Microsoft Shared\web server extensions\12\Logs

If you receive the following error message in the PSConfig log, you are experiencing this issue:

Exception: System.ArgumentException: The object with id SOME-RANDOM-GUID does not exist in the configuration store. The object may have been deleted by another operation.
at Microsoft.SharePoint.Administration.SPConfigurationDatabase.DeleteObject(Guid id)
at Microsoft.SharePoint.Administration.SPConfigurationDatabase.DeleteObject(SPPersistedObject obj)
at Microsoft.SharePoint.Administration.SPPersistedObject.Delete()
at Microsoft.SharePoint.Search.Administration.SPSearchServiceInstance.ProvisionDatabase()
at Microsoft.SharePoint.Search.Administration.SPSearchServiceInstance.Provision()
at Microsoft.SharePoint.PostSetupConfiguration.ServicesTask.InstallServiceInstanceInConfigDB(Boolean provisionTheServiceInstanceToo, String serviceInstanceRegistryKeyName, Object sharepointServiceObject)
at Microsoft.SharePoint.PostSetupConfiguration.ServicesTask.InstallServiceInstances(Boolean provisionTheServiceInstancesToo, String serviceRegistryKeyName, Object sharepointServiceObject)
at Microsoft.SharePoint.PostSetupConfiguration.ServicesTask.InstallServices(Boolean provisionTheServicesToo)
at Microsoft.SharePoint.PostSetupConfiguration.ServicesTask.Run()
at Microsoft.SharePoint.PostSetupConfiguration.TaskThread.ExecuteTask()

To work around this issue, follow these steps:

1. Do one of the following:

• Delete the Windows SharePoint Services 3.0 search service database in the Windows SharePoint Services 3.0 SQL instance. The database name starts with "WSS_Search_" and ends with the server name.

• Reprovision the Windows SharePoint Services 3.0 search service database by specifying a nondefault Windows SharePoint Services 3.0 search service database. To do this, use one of the following methods:

Method 1
Run a command-line that resembles the following command-line:

%ProgramFiles%\Common Files\Microsoft Shared\web server extensions\12\bin\STSADM.EXE" -O SPSEARCH -ACTION START -DATABASENAME “DBNameExample"

Method 2

a. Click Start, point to Administrative Tools, and then click SharePoint 3.0 Central Administration.

b. Click Operations, and then click Services on Server under Topology and Services.

c. Verify that the Windows SharePoint Services Search service is stopped. If it is running, stop the service.

d. Click Windows SharePoint Services Search under Services.

e. Under the Search Database, change the database name to any name other than the default name, and then click OK.

f. On the Services on Server page, click Start to start the Windows SharePoint Services Search service.

2. Run the following command-line:

%ProgramFiles%\Common Files\Microsoft Shared\web server extensions\12\bin\ PSConfigUI.exe

• The hotfixes for the following issues are included in this security update. However, if you install a Windows SharePoint Services 3.0 service pack after you install this security update, the following hotfixes are lost.

• You rename a Windows SharePoint Services 3.0 site collection. If you add a new user to the site collection, the welcome e-mail message that is received by the new user contains the old site name.

• You use the Application Definition Designer tool to import a database model that is included in the Business Data Catalog Definition Editor tool (BDC tool) of the software development kit (SDK) for Microsoft Office SharePoint Server 2007. After the database model is imported, the database connection string is blank.

• When you run SPWriter.exe, the program closes unexpectedly with an error message in module Oleaut32.dll. This issue was resolved by a hotfix that moved the GetTimeZoneMoveParameters() function to the end of the Owssvr.dll file.

• You migrate user accounts to a new site collection by using the Stsadmin.exe command-line tool. When the new site collection is crawled, you receive event errors 6482, 6875 and 6482 in the Application log. This issue may occur when the user's SID in Windows SharePoint Services 3.0 is invalid.

• You use a custom template in a Windows SharePoint Services 3.0 site collection. If you create a new Web page, all the standard Web parts may not be available when you click Add a Web Part.

• When you apply this security update, some third-party services may be incorrectly stopped. If this issue occurs after you apply this security update, you must restart the computer.

• You connect to a Windows SharePoint Services 3.0 site collection. If the page contains a Calendar Web part that is set to Calendar view, the page may not load completely. This issue may occur if you do not have the permission to read the Calendar list.

Prerequisites

There are no prerequisites to apply this security update.

Restart information

In certain circumstances, a restart of the computer may be required. If a restart is required, follow these steps:

1.Restart the computer.

2.Run the "SharePoint Products and Technologies Configuration Wizard."

3.Verify that all the SharePoint services are now running in the services console.

4.Verify that all the Web sites are running in the Internet Information Services (IIS) Manager.

Removal information

After you install this security update, you cannot remove the update.

If you as the SMS Engineer also own the SharePoint servers and sites then you may not have that much work to do on this one, but it is unlikely that you don't, then this might cause some concern for you.  You know that you have to get this patch out tonight, and you have to build a query for several products now, find out the owners of those servers, contact them, discuss this with them, come up with a plan for patching the server, and do this for each one, unless the one person owns all the SharePoint servers.  And when you are done with that, you get to setup a SharePoint server and test the patch on it.  So what time is it?  Do you have SharePoint on the network?  Know where the media is?  If you read each one of the articles in full, that alone is a couple hours, and that is only the first step in a very long list of procedures for deploying patches.

If you are a manager reading this, my advice, first, never ever utter the last example to an SMS Engineer because if your guy or gal has stuck it out this long, you are unlikely to find someone else as good or better for what pay you are giving them, second, on Wednesday morning bring coffee, tea, Diet Coke, your engineers caffinated beverage of chose, and lastly, relax you have a firewall and it is configured correctly - right?Thinking

 

This is the Daily Ramblings of an (ex)SMS Engineer...

Regards,
Anthony

Anthony Clendenen | Solutions Engineer | 1E Inc.


image002

http://configmgr.com

© 2007 Anthony Clendenen

Comments

  • # On Wednesday, October 10, 2007 8:47 AM myITforum Newsletters said:

    myITforum Daily Newsletter October 10, 2007 Articles Forums Blogs Wiki FAQs Email Lists In this issue

  • # On Wednesday, October 10, 2007 10:49 AM spruitt said:

    Very true, Anthony. This is part of why and patch process MUST include adequate time for testing the updates on test lab servers and pilot tester workstations... and include time for the appropriate engineers or users to make sure nothing critical was broken. That still will miss some conflicts, but at least it greatly reduces the risk.

    At my last job, I considered ten days to be the minimum time period for patching workstations under normal circumstances. We could and did deploy emergency patches in 48 hours, but at a greatly increased risk and with additional disruption.

  • # On Wednesday, October 10, 2007 12:28 PM aclendenen said:

    Thanks for the comments Steve.

    I know some very large companies with more than 100k seats that have a less than 24 hour SLA patch policy for any patch that is designated as critical.  99 out of 100 times this never causes a problem but it does require quite a bit of work by several (or more) people on Tuesday's.  This type of work is right up there with email management but is less visiable, the usres not only don't notice if the patches are installed but usually would prefer that they were not, where with email, not only do they notice but complain very loudly if they do not get it.  Email outages are less costly than a subnet outage though, work can still be completed by most without email for an hour or two, but when a subnet or domain goes down you not only lose email but most people cannot work or are very limited in the work that they can do.  So there is a constant tug of war over getting the patches deployed very quickly in this pre-0-day world and good patching practice.  Hopefully, there is a balance and some common sense when it comes to patching policies, along with some flexibility.

    Regards,

    Anthony