The hairline shows how long I've been in IT
Many people think that patch management is just about deploying the patches, and doesn't require any further thought by anyone. If your company does that, it's leaving itself exposed to unnecessary risks. Patch Management means managing the entire process. In some ways it's like the way you manage introducing a new version of an application into your company, with multiple steps of testing. That involves much more than deployment, too. Remember that patching involves updating the operating system and/or core applications in every workstation and server, every month. There are many opportunities for risks that must be managed to properly protect your company.
The Risk of the VulnerabilitiesThe first step in the management process is evaluating the risks to your organization from the vulnerabilities being patched. Someone, usually senior members of the corporate security team, should determine the real risk each presents to your particular environment. Other parts of your protection, such as firewall rules, might greatly reduce certain risks. Traveling laptops may increase certain risks. The existence of known exploits and other characteristics all come into play. The end result of this evaluation is usually a score for each update that indicates how urgent it is. That, in turn, may affect your deployment plans and flexibility to reschedule around any problems you detect.
Updates from other vendors, such as Sun and Adobe, must also be evaluated. Those updates are usually handled separately and differently from the Microsoft patches.
The Risk of the PatchesApplying the patches may present risks. Patches may conflict with your operating system configuration, hardware drivers, or applications. The impact might be minor or severe. Our company had a half dozen conflicts with applications during 2006. The conflicts with the greatest impact affected two computers, each running extremely critical functions.
Proper Patch Management includes testing as many potential sources of problems as possible prior to general deployment. That allows conflicting updates to be removed from the general deployment until the conflict is resolved. My blog post Patch Testing goes into this in geater detail.
Testing patches on servers is another separate issue. The impact of a conflict can be very high, so you need separate test servers that are updated first, then tested by the appropriate support teams. We have had frequent updates to Exchange, and at least one had the potential of disrupting users of external devices like Blackberry and Treo if the Exchange team didn't take the proper actions. These things must be identified before general deployment to properly protect your company.
Patching Success StandardsWhat level of patching success are you aiming for? Is it 99% of machines with healthy clients, 100% of all machines in use, or something else? You might well have different standards for workstations and servers. What about the status of patches from, say, three years ago? Do you need a higher success rate on higher-risk vulnerabilities? As with anything else, higher goals require additional resources. The last one percent may require more man-hours than the first 99 percent. Where to draw the line is not at all easy.
Re-applying Past UpdatesSometimes updates from past months need to be rescanned and reapplied to selected computers. The most common cause is when a new application is installed or an existing product is upgraded. A new set of updates may be necessary. You need to manage this process, removing superseded patches and assuring that these baseline updates are run at appropriate intervals and times. It might make sense to force this to run after certain upgrades or installations. This ties back to the definition of your patching goals.
Deployment Follow-upWhat percentage of your machines were patched and rebooted successfully? Why weren't the others, and what are you going to do about it? This also ties back to the definition of your patching goals.
Patching New MachinesHow are security updates applied to a newly-imaged machine? Do you depend on the baseline patching process, or do you have a script that runs them automatically? If the former, can you make that happen right away? If the latter, are you sure that script is being updated properly each month?
Patch SchedulingGiving users a longer time to apply the patches and reboot reduces disruption at the cost of accepting the vulnerability risk for a longer period. You can allow some users more time than others to reduce that risk impact, but this requires significant effort to maintain the accuracy of the Extended Schedule machines.
What day and time you release updates for pilot testers and general deployment, and when is the deadline, are important to managing the disruption vs. prompt updating.
Updating servers requires additional scheduling concerns. You don't want to patch and reboot both machines in a cluster at the same time, and you probably shouldn't reboot your SMS servers while they are deploying patches to other computers. All server updates typically must be performed during the maintenance window, so you may need many separate updates during that time to do selected groups of servers in each interval.
Maintaining Deployment DataIf you use pilot testers, you need to spend some time each month updating those lists. Employees replace their computers, other employees transfer or leave the company and must be replaced as pilot testers. There are many types of deployment data that may require review and updating, including server patching schedules, Extended Schedule workstations, etc.
Updates to Other ProductsDo you know what to do if Sun announces a vulnerability in a version of Java? Can you tell which machines have which version of Java installed? Do you know which applications require which version, and may not work with replacement versions? Do you know which of your installer packages contain Java?
That can easily be the worst case for updates announced by other vendors, but proper management of patches and risks requires addressing all such announcements.
Technology ChangesDo you know what the next version of SMS will change in handling patches? If you use other tools as part of your patch management process, are new vesions being released? Are other vendors releasing products you should consider? Somebody in your organization needs to be aware of these changes and alerting people of ones that deserve closer study.
Impact on other DepartmentsDoes your patching process require significant effort by the desktop and server support teams? If so, you might be able to reduce that and provide a tremendous benefit to your company.
As you can see, there are many aspects to patching besides the simple monthly patch deployment. Which ones apply to your environment will vary. Ultimately, patch management is a form of risk management. Performing the proper activities helps to manage the risks, and also use the resources effectively.
I've seen several postings recently from people just getting into SMS patching that don't know