Steve Pruitt at myITforum.com

We apply patches to protect our computers from outside attacks, but we also need to protect our computers from the patches themselves. This isn’t just idle speculation – last year our company had a half dozen proven instances of conflicts between MS security updates and applications.

We need test plans that help identify such problems early enough to permit timely resolution and changing the deployment plan to minimize the risk of disruption. We can rarely, if ever, catch all such problems during testing, so we also need established procedures for identifying and handling conflicts that are discovered during or after the mass production deployment.

Types of Conflicts

• Common Operating System Components
  A conflict with the base OS should be extremely rare, but if your computers are configured to include features or options that are not generally used there could be a conflict. Your first testing should be with computers that have the common base configurations to detect any such problems. Note that problems may not be seen unless the affected component is actually used – you need more than a successful reboot to be reasonably certain you don’t have such errors. This is most likely to be discovered when the first computers in production use are patched.
• Drivers
  Updates may cause a problem only on computers with a particular hardware configuration or driver installed. The past reports of such have never had an obvious connection to the patch description, so always be aware of this possibility during deployment. The best way to detect it is through pilot testing that includes a diverse group of users and workstation models.
• Applications
  Any application could be affected by patching activity. Particularly watch for patches that update components you know are used, such as IE updates, but surprises are common.
• Application Installations
  An existing install file, especially an msi built by capturing the vendor install, might require permissions to files or registry keys that are changed by a security patch.

Testing Plan

I recommend a phased approach designed to detect conflicts as early as possible.

First is basic testing to be sure the patches are detected and execute as they should, and the computers reboot successfully, in your standard environments. I prefer to test with virtual machines that can easily be reverted for further testing. I used machines with Windows 2000, XP SP1 with Office XP, and XP SP2 with Office 2003. After rebooting, make sure that major applications like IE or Office still open and work.

Second is patching a few real computers in normal use. I followed Microsoft’s “dogfood” principle, and patched the computers used by the patch management team. We knew best how to recognize abnormal behavior in the patch deployment, and could exercise core functions thoroughly and quickly.

Next is patching one or more sets of pilot testers. We use two sets, Alpha and Beta pilots, representing two and eight percent of the installed computers respectively. In order to get thorough testing, the department managers must be involved in deciding which functions to test and thus which people to use as testers. One of my fears is that a patch might break an application like RightFax, that is used by a large percentage of the employees but probably not on any department’s critical application list. It’s critical to make sure that any detected problems are analyzed quickly to determine if they are really caused by patching and identify which patch was responsible.

One final aspect of preparation is monitoring discussion groups such as PatchManagement.org to learn about problems other people find. Remember to take isolated reports with a grain of salt, but don’t completely disregard them. Multiple reports should be cause for concern.

Problems Discovered After Deployment

Sometimes problems aren’t detected until after the full production deployment. As I said earlier, we had a half dozen such instances last year. The first step, as during pilot testing, is determining if patching is really responsible and identifying a particular update. I prefer a process of uninstalling one patch at a time, rebooting, and retesting. Once the failing function starts to work, reapply the last patch removed and reboot again. If the problem comes back, you then know the cause.

OK, now what? You need to do four things:

1. Protect the affected machines by removing the offending patch and putting the machines into a collection that prevents the patch from reapplying later.
2. Develop a plan for protecting the computers from attack based on the unapplied patch.
3. Develop a plan for resolving the problem, through changes either by Microsoft or the application vendor.
4. Notify the appropriate IT and Security management of the incident and your remediation plans.

Of course those steps may be altered by the magnitude of the problem. If it affects most or all of the company, you need to start by notifying management and security.