Steve Pruitt at myITforum.com

ConfigMgr OS Deployment

2008-01-13T18:32:00Z

As I promised earlier in the mssms mailing list, I just created a set of Wiki pages to begin documenting the ConfigMgr OS Deployment process. This includes a general introduction page with sections for various areas of interest plus a detailed description of the process and procedures I developed in recent weeks for creating a reference image. There are many sections that have not been completed. I hope other community members will jump in for those.

When reading these pages, keep in mind that they were developed from my initial experimentation in a test lab. I fully expect that they contain errors and misunderstandings, that I hope community members will point out and correct. Also, the security aspects are based on a lab environment that no one else could access. In real world use you should create and reference suitable accounts for image capture and generally assure proper access controls.

The one known problem with the detailed instructions is that software updates do not apply successfully before the image is captured. I'll update the instructions once that is resolved.

Desired Configuration Management - using Configuration packs

2007-12-29T01:43:00Z

Microsoft and several vendors have created a variety of DCM configurations, or rule sets, checking for various security requirements. These include basic checks for Windows 2003 and SQL 2005, and requirements for SOX, HIPPA, and other sets of regulations. These can provide a great basis for checking the status of your organization's environment without the need to completely create the appropriate definitions. You can see the complete catalog of configuration packs here.

The following comments and suggestions are based on my testing of a few packs created by Microsoft. Most of the comments probably apply to packs from other vendors as well.

The Microsoft packs are downloaded as MSI files. When installed they create an entry in Add/Remove Programs and a folder in Program Files containing one cab file. After that cab file is imported through the Configuration Manager console the entry in Add/Remove Programs is not needed and can be removed. I have no idea why they don't just let you download a cab file directly.

To import the definitions:

In the Configuration Manager console, right click on Configuration Baselines and select Import Configuration Data
Click Add and navigate to the desired cab file
Click Run, then Click Next
The Summary will show what Baseline and Configuration Items will be added
This is the first time you find out if the pack contained a Baseline as well as one or more Configuration Items - many only contain CIs
Click Next, then Close to complete the import operation
If the pack did not contain a Baseline you will need to create one so the rules can be used. The process is sufficiently simple that I won't go into it here.

The next step is to examine the properties of the Configuration Items to see what is being tested, then advertise the new Baseline to a small representative sample of computers to see the results. In my testing, two of the Microsoft Configuration Items tested for file versions that were already upgraded. The tests required an equal match, so errors were reported for machines that had newer files. If you run into this, see my previous blog article for instructions for editing the incorrect CIs. Think carefully about what file versions you want to test for, to properly reflect the desired service pack and update levels.

The standard Configuration Manager DCM reports include many compliance reports. As with most such reports, you can get a high-level summary report and drill down to details of which exact validation tests failed for each computer. Analyze these reports carefully and decide on a plan for correcting the deficiencies reported. You can also see detailed compliance reports for any computer through it's Control Panel Configuration Manager applet by selecting the desired report in the Configurations tab.

In many cases the remediation will require applying a service pack or selected updates to all applicable computers. Those can be handled efficiently through Software Updates. If some errors are reported for just a portion of the applicable machines, but enough of them to warrant automated solutions, you can create collections based on the DCM results and distribute applications or scripts that will apply the required changes. The WQL queries you can create can not reflect the lowest level of detail, of specific validation tests. The finest detail is represented by the list of rules shown in the Configuration Manager console under each Configuration Item. These rules often contain one or very few validation tests, but some may contain several dozen or more. The scripts or applications you advertise should allow for this and only make the updates required on a particular machine. In extreme cases, you may need to use the editing capabilities described in the previously-referenced article to divide the validation tests into two or more rule sets that can be addressed individually for remediation.

For more details about creating remediation collections, see How to Remediate Non-Compliant Computers Using Software Distribution. If you have problems with any of this, see Troubleshooting Desired Configuration Management Issues.

Updating Desired Configuration Management Baselines

2007-12-28T01:47:00Z

I've been testing the new Desired Configuration Management feature, and I see it has fantastic possibilities for helping to manage a company's infrastructure. However, I also quickly found one problem. If you use the standard Baseline Configuration Packs that can be downloaded from Microsoft, you'll find that some of the file version tests are obsolete. I ran into this for Windows 2003 DNS and Basic SQL 2005. Fortunately, we can update these definitions to reflect the service pack and update levels we want. The process is documented at TechNet, of course, but I had trouble following it at first. You can't directly edit the configuration data you download and import into DCM. If it's not created at your site it's read-only. The solution is to create duplicate versions of the components you want to edit, edit those, then substitute them in a duplicated baseline. Note: sometimes you will be downloading configuration items and creating your own baseline. In those cases, you don't need to duplicate the baseline, only the desired configuration item. Here's my step-by-step process that worked:

The first step is deciding which tests you want to change. You can use the following steps to examine each rule and research them, but I prefer the simpler way... run the selected configuration tests against a selection of typical machines, then analyze the errors that are reported. Some will really be things you want to correct, others will reflect obsolete tests that need to be updated.

Here's the steps to edit the baseline and validation rules:

Expand the list of configuration baselines
Right click on the baseline you want to edit and choose Duplicate
Enter a new name for the local copy of the baseline and click OK to complete creating a duplicate copy
Expand the Configuration Items section and select the rule you want to edit in the right hand pane
Right click on the rule and select Duplicate
Enter a new name for the local copy of the rule and click OK to complete creating a duplicate copy
Select the new copy of the rule, right click on it, and choose Properties
Select the Objects tab
Double click the object you want to edit, then select the Validation tab
Select the desired test (there's usually just one) and click Edit
Revise the test as desired. In my example, I changed it from 'Equals' to 'Greater than or equal to'
Click OK to close window
Repeat as needed for other objects in this rule
Click OK to close remaining windows
Select Configuration Baselines in the SCCM Console tree
Right click on the desired baseline (local copy if you made a duplicate earlier) and choose Properties
Select the Rules tab
Click the appropriate underlined configuration item category, such as "applications and general" as shown above to display a list of all available rules of that type
Select the newly edited rule to add
Click OK
In the Rules window, select original version of rule, click Delete
click OK to close the window
If you duplicated the downloaded baseline, now you want to make sure you don't run it any more. You can do this in several ways:

Remove the association with a collection:

Open the properties of the old baseline
Select the Assignments tab
Select the collection(s) and click Delete

Disable the baseline by right clicking on it and choosing Disable
Delete the baseline by right clicking on it and choosing Delete

If you replaced the original baseline, remember to assign the new baseline to the desired collection(s).

Test your changes on a small pilot group before assigning it to a broad collection, of course. If you used a pilot test to discover the original problems, the same collection is ideal for testing the changes.

Deploying applications as software updates

2007-12-26T15:23:00Z

Have you ever needed to decide how to deploy an application that requires a reboot? Has a manager ever asked if you can deploy it as a software update, using the same notification and scheduling options? I've faced both, and I'm sure most SMS admins have as well.

SMS doesn't lend itself to deploying applications that require a reboot. There are no notification and scheduling options available, as there are for updates, and the update process is designed just for security updates from Microsoft. You need to either set up the program to only run when no user is logged on or wrap the installation in a custom script that provides the needed capabilities.

This can be much easier under Configuration Manager, if you also install System Center Updates Publisher (SCUP). Updates Publisher extends the software updates process to support update catalogs from other vendors and also custom updates created by the SMS Administrator. You could create a custom update to deploy your application installation that requires a reboot. SCUP updates are completely integrated into Configuration Manager, and are deployed exactly like other updates using the same capabilities. The MyITForum Wiki article about SCUP provides some additional information.

The basic idea is to create a custom update rule that will install the application, and deploy it to a collection of target computers. It's not quite that simple, but close. There are a few key things you need to do.

First, prepare the installation as an exe or msi if it's not already in that form. SCUP only supports exe, msi and msp files. If you use an exe, document the return codes that indicate successful completion and those that indicate a reboot is required. Test to be certain your command line results in a successful silent install as a new install or an upgrade.

Second, you need to plan the rules you will use. An SCUP definition uses three rules (all are required):

Prerequisites: these are things that must be present (or absent) before the software can be installed. You can test for the operating system, files and versions, registry info and WMI query results.
Applicability: files, registry info or WMI queries that make the update applicable. For an application, you might use something generic that all computers would satisfy.
Installed: files, registry info or WMI query responses that indicate the application is installed. If the installation uses an msi, the new rules wizard will automatically extract the installation code and create a rule for that. If you create your own rule, be sure it will accurately reflect failed installations and if the application is uninstalled. This rule prevents the application from being reinstalled every time updates are reevaluated.

Keep in mind that after the custom update is published, it will be part of the WSUS database for all machines. All computers will scan for this being applicable, and that status will be included in software update reports. This will only be installed on computers included in collections the update is deployed to, no matter how many others report it as applicable.

Once the update is published, it is deployed exactly like any other update. You would create a deployment selecting the one update, choosing the desired notification and scheduling options and targeting the desired collection. This will require careful thought and testing to be certain everything works as expected. For example, if regular update deployments specify the option to install any other updates due within a certain time period, this might include your custom update and cause it to be installed before you expected.

Since this is deployed as an update, normal software deployment reports don't apply. Before using this capability, be sure you understand which software update reports you will use to monitor and manage this process.

The same concept might be used with SMS 2003 and ITCU (the predecessor to SCUP), but ITCU was rarely used. This is partly because ITCU was not integrated into the SMS deployment process as well as SCUP is with Configuration Manager.

Reboot before patching

2007-12-20T18:28:00Z

We've all experienced users that blame absolutely everything on patching. My personal favorite is the person who blamed us for the problem that started just after she received my email announcing that patches would be applied the following week.

Many of the reports are more reasonable, because the problem really did start when patches were applied. This is frequently because some other change had file operations pending the next reboot, and many users only reboot when patches are applied. There's no way the users can distinguish those results from ones really caused by patches.

If this is a significant issue for your organization, one easy solution is to schedule a reboot of all workstations a few days before patches are deployed. You should exclude any that are in a special-handling collection, of course. That should make most unrelated issues turn up before patching starts, making them far easier to diagnose. This can also reduce the problem of SMS 2003 patch deployments causing a reboot as soon as the advertisement starts, as described in my earlier article on that subject.

Configuration Manager (SCCM 2007) Software Updates

2007-12-17T01:43:00Z

I just finished writing the initial Wiki articles about Software Updates in Configuration Manager, and hope others will contribute to it. It covers migration from SMS 2003, installation and configuration, deployment design, reporting, suggested normal monthly activity schedule, and troubleshooting. Of particular value to anyone who has begun testing this process is the section on deployment options in the deployment design section. This lists all of the major options you can select, describes how they work, and alternate places where they can be selected.

One intended section remains to be written, about updating servers. The primary feature that affects server deployments is the ability to restrict updates to maintenance windows specified by collection. That feature is already described in the deployment design section, so I'm trying to decide what else to say about updating servers.

If anyone sees errors or thinks of other topics to be covered, feel free to dive in and edit the material - or comment to this post or write to me.

Patching in SCCM 2007

2007-11-20T22:44:00Z

Here’s a summary of some of the most significant differences I found in patching between SMS 2003 and SCCM 2007. Note that other differences I don’t list may be significant to your deployment process. It’s vital to thoroughly test this and plan the changes you’d make to your environment. Also, these reflect the results of my testing; your results may be different. Also, read Chris Mosby's blog, where he offers a preview of Chapter 14 of his book about SCCM. It describes the Software Update process in SCCM

ITMU becomes WSUS (plus more)

Greater variety of updates can be deployed

ITMU supports nearly all security updates
WSUS supports all updates that are in Microsoft Update
That, in turn, means you need to pay closer attention to what’s newly released. It may include non-security updates that you don’t want to deploy.
This also makes it much easier to deploy these non-security updates, if desired.

Install ITCU to be able to deploy non-Microsoft updates from hardware vendors, Adobe, etc. It’s fairly easy to add your own upgrade deployments for any applications or products.
The new Desired Configuration Management capabilities let you use patch scanner-like analyses of all sorts of characteristics; this can be used in conjunction with patching or independently

Scanning

Synchronization with Microsoft Updates happens on a specified schedule and can be run on demand
SCCM does not allow you to schedule scans separately by collection; there is one global schedule for the system
Scans are initiated by the client as part of various activities, along with the scheduled scans
Scans run at random times during the two hours after the machine policy is received, to spread out the load on the network and servers
There is no apparent way to schedule scans on specific dates, so the only way to assure accurate patch requirement reports at the beginning of a patch cycle is to run scans daily
It is easy to change the scanning schedule, so you could easily scan daily for the first days of a patch cycle and use some longer interval the rest of the month
There is a separate schedule for rescanning based on updates that were scanned previously (i.e., no attempt to find other updates to scan for)

Deployment Scheduling

You can still deploy with different options to different collections
It’s now much easier to use staged deployments:

Create an Update List of the updates you want to deploy
Create multiple Deployments targeting the desired collections with the desired options
Use Deployment Templates to assure consistency
This can include a mix of interactive and silent deployments with different schedules
Only one Deployment Package is created and distributed to the DPs

In SMS 2003 you can create a package that allows postponement for a specified time from either the Time Authorized or the Time Available
In SCCM the deadline is only based on Time Available, which is the “advertisement” start time

Baseline Patching

SMS 2003 requires that you have a separate deployment package of updates from past months with a separate advertisement as the most efficient way to rescan for past updates and reapply them as required
SCCM has a rescan schedule that is distinct from the regular scanning schedule
SCCM rescanning can use the original update deployment packages; there is no need to combine them
SCCM software updates will get update executables from any available package; that means you could combine updates from previous months into rollup packages for easier management if desired
Many other ways of combining past and/or current updates are possible, to accomodate any desired update management practices
Keep in mind the effects on network traffic and DPs of adding or moving updates between packages - make sure your plans will be effective and efficient at all levels
Also remember that the most valuable reporting may be based on the deployment structure

Migration

You have to uninstall all scanners except ITMU before upgrading from SMS 2003 to SCCM 2007
If you do a side by side upgrade you can install ITMU under SCCM 2007 to support SMS 2003 client
If you can’t complete updating all clients from SMS 2003 to SCCM 2007 between two patching cycles, keep one SMS site server operating in case you need to deploy any ESUIT updates. These can not be deployed through SCCM 2007.
Install WSUS 3.0 before SCCM, and cancel out of the wizard without configuring it –SCCM will configure WSUS when that site role is installed

User Interface

The appearance of the balloons and dialog boxes, and the steps needed to run the updates now or a specified later time are different
If an update does not require a reboot, there’s no dialog box saying the update has completed as there is with SMS 2003
The deployment process and schedule at your company may change if you take advantage of new features
All of these changes must be communicated clearly to users shortly before your first SCCM update deployment to minimize confusion and calls to the Help Desk, and the disruption resulting from such confusion
If there are file changes pending a reboot when new updates are distributed, the user is informed in the regular dialog box - the system does not force the reboot until the patching deadline (note: I’ve heard reports of other behavior, but my testing has been consistent)

Reporting

I found it frustrating not having the advertisement-based reports and queries for scanning and updating that I was used to
The new reports are based on the current patch status of each machine, not the status of the scanner and deployment executions plus patch status
Be prepared for this, and think carefully about what reporting you’ll need
There are many new reports – study them before creating your own, you may not need them after all!

Patching and Communications

2007-10-21T23:47:00Z

Patch Management is only partly a technical activity. Good communications are at least as important, if not even more so. No matter how well you designed your deployment plan to support the needs of your business, you need to communicate effectively with the right people each month to be successful. The general principle is to avoid surprises. If everyone knows what’s going on that concerns them, the whole process proceeds smoothly. What things should you communicate, and who do you tell? The answer always depends on details of how your company is organized, how it operates, and how you deploy the updates. Here are some examples to consider:

Anticipated Patching

What: On Thursday before Patch Tuesday, Microsoft announces the updates they expect to release the following week. Keep in mind that the actual release may contain more or fewer updates and could also include re-released updates.

When: Shortly after the pre-announcement is available.

Why: This announcement doesn’t contain a lot of detail, but it’s enough to have some idea how great the impact may be.

Who to tell: Share a summary of this announcement with everyone involved in Change Management, so they can begin thinking of possible impact on other planned activities.

Patch Release Details

What: Summary of details of each update, including the affected operating systems or products, how the vulnerabilities could be exploited (i.e., through email, web sites, network connection or physical logon), if the update requires a reboot, and if it can be uninstalled.

When: On Patch Tuesday afternoon or first thing the following morning.

Why: This helps people understand what applications might be affected and the impact of the deployment.

Who to tell: Everyone involved in Change Management or testing of server applications.

Deployment Schedule and User Impact

What: The planned schedule for deploying updates to pilot testers, other special groups, and the general population, including the deadlines for each group that is allowed to postpone or schedule updates. Explain the normal activities required and any unusual activities or appearance. Include a link to a comprehensive explanation of the patching process for users.

When: The afternoon after Patch Tuesday, or as soon as the schedule and activities are known. This should be after initial testing is completed.

Why: If users know what to expect to see and what to do, you’ll have fewer calls to the Help Desk and fewer problems. This also serves to explain the patching process to new employees. This also helps many people schedule changes or other activities to avoid conflict with the updates.

Who to tell: All employees that will see update activities. You may want separate communications to pilot testers and other users that have different schedules or activities than the majority of users. That allows simplifying the message to all employees and reduces confusion.

Special Deployment Announcements

What: These are announcements to pilot testers, users responsible for exception machines that get non-standard deployment options, and anyone else who sees deployment schedules or options that are different from the majority of users.

When: On the day updates are deployed to each group, or very shortly before.

Why: These announcements serve as a reminder of the update schedule for their particular machines, a reminder of which machines are affected, and a reminder of the testing and feedback that is expected of them. This is intended the improve the quality of pilot testing and feedback, and reduce problems resulting from updating exception machines.

Who to tell: All users who will receive deployments on a different schedule than the majority of users or who have different options for postponing or processing updates.

Deployment Issues

What: You will occasionally find issues with certain updates that change the deployment plans. These might include problems that prevent proper patching or conflicts with important business applications. The information you supply includes the affected update, the issue preventing normal deployment, the affected machines, the revised deployment plan, and the plan for resolving the issue.

When: As soon as the issues are found and the revised plan is decided.

Why: The affected people need to know if an update will be deployed separately, later, or if the entire deployment is being delayed, or if an application is affected by an update.

Who to tell: In each case, the Information Security and appropriate IT management must be informed. Other people may need to know if they have to take different actions because of these changes. Any more general announcements must be written and timed to minimize confusion that can result.

Patching Progress Reports

What: Management reports of the progress applying updates and explanation of any differences from normal.

When: The management receiving these reports will specify the desired frequency. In larger organizations there may be separate detail reporting to lower levels of management and summary reporting to senior management, with different frequencies. Typically there are a small number of regular reports at key dates, plus exception reports if required.

Why: IT and Security management will generally want progress reports for any important projects. Security patching, even though it occurs every month, qualifies in most companies. In addition, if problems arise in any portion of the deployment it’s vital to inform the proper people as soon as possible. For example, an update to a particular product may frequently fail for some reason that didn’t turn up in initial testing. This would require developing a plan for rerunning that update, which may entail unplanned user disruption.

Who to tell: This normally includes the two levels of management above the person managing the deployments plus the Information Security manager. In some organizations, management of server support, desktop support, division CIOs, and others may want to be included in such reporting.

You have to tailor this to your organization, of course. The key is to remember that patching affects every computer, every user, in the company. If the users and managers understand what will happen and when, everything is likely to go much smoother.

Patch Process Design and Communications

2007-10-21T23:32:00Z

Have you ever designed a patching process, only to find that users and managers complained after you implemented it, and no one supported you? Good planning and communications with the right people help assure that you develop a good plan, that the plan is approved by everyone concerned, and minimize problems and complaints during the deployments.

Before designing a patch deployment methodology you must understand your tools and objectives:

Understand the options available to manage the deployment. These are different for WSUS, SMS 2003, ConfigMgr (also known as SCCM 2007) and other software. If you use SMS, you can combine the patch deployment options with other SMS capabilities such as running only if no user is logged on, wrapping the deployment execution in a script that adds options, etc.
Get your management’s guidance and expectations. What are your trade-offs between security and disruption? What standards are expected for completing the patching? The best security means completing the patching quickly, but that also requires the greatest disruption. Avoiding disruption extends the time needed to complete securing the network. This should be one of your first questions to your management. The preferred guideline is in the form of “Use the least disruption that allows completing patching to 99% within ___days” or something similar.
Understand the business needs of your environment. That means talking with key business unit and IT management.

Your plan has to include a few key parts that you can develop through communications:

What are the restrictions on rebooting computers in various groups? Some can be scheduled, some require control by the computer users or other responsible people.

Most companies have some groups that can be rebooted any night without problems, others that can never be rebooted during the night, and others that need to control the timing. For example, if you have a 24-hour call center, you can't do anything that will reboot all of those representatives at the same time. It needs to be spread out and controlled. Operations consoles can't be rebooted during their working hours, which are often the middle of the night. There are usually some IT managers, such as Desktop Support or Call Center, who can identify various such groups and the managers there to talk with.
Will reboots occur immediately after the updates are applied or some time later? Will they be automatic or manual? If manual, how will you manage and enforce the requirement?
How much time do various group need to schedule reboots? For example, Insurance company actuaries often run calculations that require several days to complete. Do you have any similar functions?
How will you handle computers, such as laptops, that are offline during the normal patching period? Will they be patched immediately after they reconnect, or will users be allowed a period to complete this?

How will pilot testers be selected, and replaced as required? Who will test for a department if a pilot tester is on vacation?
How will you identify the computers that fall into different groups, and how will you maintain the accuracy of those lists? That includes groups that are treated differently for deployment schedules or options and pilot testers. Maintaining the accuracy of these lists is vital - errors in these lists will cause the biggest problems and complaints most of the time. If possible, find a way to identify critical groups of computers automatically by subnet or some other common characteristics that SMS can use in collection queries.
Who do you need to communicate with on a regular basis? How will those lists be maintained? These may include users that will get different deployment handling than most users, pilot testers, managers of key departments, etc.

Find out which departments are particularly critical to the business, and talk to them to find out if they have any special needs. Find out which departments are known to complain to IT senior management, and include them in your planning from the beginning. You need to come up with a patch management system that will meet the needs of all of these groups. Talk to each group and find out their requirements. Explain the importance of protecting their computers with the minimum of disruption.

Once you develop a plan that you think will meet their needs, you have to convince a number of groups of people that your plan is sound. That means selling it, and also listening closely to all complaints or objections. In most cases you should be able to explain how the issues are handled in the plan. Sometimes you'll need to think about the issues the raise and revise the plan accordingly.

The first group is your management. You can't go any further until your boss and perhaps one level higher have agreed to support your plan.
The second group is the corporate security manager or team. If they believe you have a good plan that adequately protects the company, they can help convince other people.
The third group is all of the other managers you talked with while developing your plan. Show them how your design reflects their needs, and explain their responsibilities to assure that their computers are protected with the minimum of disruption.

Finally, if you are changing what the users will see or what they must do, you have to explain this to all of them. Use all of the communications opportunities your company offers, including newsletters, regular meetings with all managers or all employees, etc. Follow this up with an email to all employees that has a brief explanation and a link to a more complete presentation for those that are interested. Ask your corporate communications staff to help prepare and send this out.

If you follow this communications process, you should have the understanding and support of the IT and business unit management that are most affected by your patching strategy. This will go a long way towards minimizing problems as you implement the plan.

Recurring advertisements are required for patching

2007-10-16T17:31:00Z

My co-worker Shaun Cassells made an important discovery when we began patching with SMS 2003. When you advertise the patch update packages, you must always set up a recurring execution. The logic of the patchinstall process requires this if users are permitted to postpone the updates. See KB842717. If you have some update packages that allow postponement and others that don't, you're best off being consistent and using recurring advertisements for all of them. That greatly reduces the chance of creating this error. If you have DSUW create the advertisements it will always create a recurring schedule.

If you have an update that you really only want to run once, such as for pilot testers, just select a monthly recurrence schedule and delete the advertisement before the month is up.

In general, it doesn't hurt to have updates run on a recurring schedule. The update will rerun the scanner, and then only apply any updates included in the package that the scanner found were applicable. Once a set of updates are applied, rerunning that package will do nothing unless changes to the computer made one of them applicable again.

Scanners should also have a recurring schedule that assures they will always be run before the update package. Although the update package repeats some scanning functions, that does not replace running the full regular scanner. The regular scanner detects updates that are applicable or installed. The scan function within the update verifies if that information is still accurate, but won't perform the initial detection.

Using psexec to install the SMS client

2007-10-01T19:23:00Z

Sometimes the easiest way to install the client may be running it with psexec. That's especially true if the machine hasn't been discovered by some mechanism. The options for doing this often cause confusion, because there are two files involved - ccmsetup.exe and client.msi. CCMSETUP looks for client.msi in whatever folder ccmsetup is run from. If you use the -c switch that's normal with psexec, to copy the file being executed, it only copies the exe. That then hangs when it can't locate the msi, and the ccmsetup.log file is filled with messages reporting that it can't find client.msi in C:\Windows\System32 and will retry in 20 minutes.

The solution is to run ccmsetup from a location containing both files. The easiest is usually on an SMS site server. The proper command line is:

psexec -s \\computer \\SMSserver\client\i386\ccmsetup.exe [ccmsetup switches]

The -s command says to run using the system account. Do not use -c, as that will cause the errors described above. Using -d is optional; ccmsetup terminates pretty quickly. I generally leave it off so I can see the zero return code.

Patch Management Process Overview

2007-09-28T19:02:00Z

I've seen several postings recently from people just getting into SMS patching that don't know where to start. This article is intended to help them understand the basics of the process, and link to various web pages that either provide greater documentation or are used in normal operations. This is based on SMS 2003 with ITMU.

Patch Management Principles
These articles explain the overall concepts and principles involved:

Getting Started
First, install ITMU on your Central Site Server. You may also need the Extended Security Update Inventory Tool, which supports products that ITMU does not support. That can always be installed later if needed. Then go through a small test of the normal monthly cycle, described below. That will help you understand what all the articles are talking about. Then you're ready to begin reviewing the peculiar needs of your business, and design a patching strategy that fits. Ask questions in the mssms mailing list when you need help.

Monthly Cycle
Microsoft gives a heads-up of what they expect to release on the Thursday before Patch Tuesday at their Security Bulletin Advance Notification web page. Patches are normally released on the second Tuesday of each month. Their Current Security Bulletins web page links to the current month summary, which in turn links to a detail page for each individual bulletin. That's usually available beginning about noon Pacific time, but that's not guaranteed. It takes a while for all of Microsoft's web servers to be updated, so you'll often find that some links don't work until later in the day. When you are getting started, you should study the bulletin summaries for a couple of recent months, and the pages for each individual bulletin, in detail. Each type of page has sections with a + in front of a header line. Click on that to expand the section. When getting started, you should expand and read every section of every page for the previous couple of months. At the end of that time you'll have some idea of what information is available where.

You should also read the current month Detection and Deployment Guidance article. The articles for each month are linked from KB910723. These tell you which updates for which products can be detected and patched through ITMU, and which require the Extended Security Update Inventory Tool (ESUIT) or other means.

Sometime during the evening of Patch Tuesday you should run the ITMU Synch program. This downloads the latest updates to your server. Schedule the ITMU scanner to run on machines later that night, so you'll know which updates are needed in your environment. There are two scanner programs. The expedited version runs hardware inventory after the scanner completes. That sends the scanner results to your server for review, and causes some amount of network traffic and server activity. If you have a reasonably large network, you might use the expedited scanner on your pilot test machines and the regular scanner on the others. That way you'll have pretty good information available first thing Wednesday morning, and better counts by the end of the day. If the ESUIT scanner is needed for your environment, the latest version is downloaded manually from the ESUIT download page. That's installed on your server, and the scanner runs just like the ITMU scanner.

On Wednesday, you review the new updates that apply to your computers. This is shown in the Security Updates section of the SMS console. Sort by the bulletin number column to make it easy to find the latest updates. Then decide how to deploy them. This allows for the risks the updates protect you from and the risks inherent in the updates. You should always test all updates on a set of pilot test computers before the general deployment, to see if they conflict with any business-critical applications. Read my Patch Testing article for more details. You can also get valuable information from the Microsoft Security Response Center Blog and the Patch Management mailing list. Remember that reports there won't always mean problems in your network, but they will give you advance warning of possible issues. Some reports in that mailing list are over-reactions, just as some of your users will blame any problem on the patches.

Then you'll create one or more patch deployment packages using the Distribute Software Updates Wizard in the SMS console, and run them on test machines. I always included my own PC as one of the first test machines. If everything looks good, run them on all machines.

The other articles in my blog explain concepts for setting up your deployment and testing strategy. They'll make the most sense to you after studying the updates from the last few months at Microsoft's web site, and working out how you'd deploy them through SMS.

Again, ask questions in the mssms mailing list when you need help. Patching is not simple, and it's an area where it's critical to be careful. Remember, when you are updating every computer in your company, there's no such thing as a small mistake. Any error can affect the security of your company or the degree of disruption you cause your users.

Allowing selected users to postpone patching

2007-08-26T19:52:00Z

Someone asked an interesting question in the mssms mailing list recently. I answered it there, and thought it was worth recording in a blog as well. He wanted to know how to deploy patches so that some users could postpone the update, while the others could not. It's actually fairly simple. What follows is even easier than my original email response.

First, create an interactive patch package with the Date Authorized in the past, and allowing the maximum postponement you want for the group that's allowed to postpone the patches. See Settings for Distribute Software Updates Wizard for details.

Second, create a second program in that SMS package that's identical to the first. Use a name includes something like "No Delay" and change the g: parameter in the command line to G:1. That allows one hour postponement from the date authorized. Since that time is long since past, this program will cause patches to be run as soon as the countdown time expires, then reboot after a second countdown.

The third step could be the hardest. You need to get the collections set up properly. First create a collection of computers where you want to allow postponement, based on whatever user-to-machine data is available to you. Second create a collection using a query that specifies all machines not in the "allow postponement" collection, and limited to your existing production deployment collection. That will end up being all machines you want to patch except those that can postpone.

Finally create the advertisements. The "Allow postponement" collection is advertised to the SMS program that was created by the wizard, and the "All other machines" collection is advertised to the "No Delay" program you created manually.

In another email in that thread, Marcus Oh suggested another solution that would be better in some environments. His idea was to create a program that checks which groups the logged-on user is in. If the user is a member of Group A, the program would then execute patchinstall.exe with the command line options to postpone, otherwise run it with the command line options no not postpone. You'd update the SMS program command line to run this program instead of patchinstall.

As always, test carefully before proceeding!

Identifying and selecting Office patches

2007-08-19T02:09:00Z

If you are preparing your first Office patches, you may have a hard time finding them in the Distribute Software Updates Wizard. Office patches are set up a little differently than OS patches. How differently depends on whether you are using ITMU or the SUSFP Office scanner. In each case, the main problem is that the patch for each affected product has it's own Q number. These are listed in the bulletin web page, but I prefer to identify them through SMS. That's especially valuable for updates such as XML that may have both OS and Office components affected.

ITMU
The SMS console's Software Updates section has all updates and can be sorted by any of the columns. In ITMU, Office patches have the Q and bulletin numbers in the respective columns. Click the heading of the Bulletin number column to sort by that data, then locate the current updates. You'll find all of the product patches together. I like to export this list to an Excel spreadsheet and sort there, then delete all rows I'm not concerned with for that deployment. That makes it easiest to make sure I locate and select all of the proper patches in DSUW.

The individual patches can be located in DSUW the same as for OS updates, by entering the Q number as the appropriate filter.

SUSFP Office Scanner
If you are using the older Office scanner, either because you haven't upgraded to ITMU or because you still have some computers with Office 2000, it's trickier. The Office patches are listed in the SMS console's Software Updates section without any data in the Q or bulletin number columns. The first step is to export the Software Updates data to a spreadsheet. Then use Find to search for each of the Q numbers listed in the bulletin web page and move all matching rows to a separate sheet. Watch carefully for Q numbers that appear on more than one line, and move each of them.

These patches are located in DSUW by entering the Q number as the filter for the Name field. Make sure you locate and select all of the appropriate patches for all products in your environment.

General
As you search for these updates in the spreadsheets or console Software Updates, make certain you identify all patches you need to deploy. In some cases they will be listed under different scanners. They might have updates for OS components, Office products supported by ITMU, and Office products that are not supported by ITMU. Be sure to make a complete list of the patches to be selected in DSUW. Always be on the lookout for variations and exceptions. Check the counts reported in Software Updates after your scanners have run on most machines and the results reported through hardware inventory. Be prepared for surprises.

I have also seen that computers with a mix of Office versions installed, such as Office XP with a different version or SP level of Front Page, Access or Power Point can mess up the scanners. That seems especially true of products from the same version but different service pack levels. That can easily happen if Office products that may be installed separately, such as Front Page, are not carefully kept at the latest service pack level, including updating the installation packages. Baseline patching can help with this, especially if it includes the proper Office service packs.

Unexpected reboots when patching

2007-08-07T20:19:00Z

Have you ever had users complain that their computer rebooted unexpectedly about the time you deployed patches? Maybe it gave them a brief warning, but no chance to postpone? We ran into that, and it took a fair bit of digging to get the answer.

We found out that one of the first things that the patch update program does, even before looking at the options you selected, is to check the PendingFileRenameOperations registry key. If there are any operations pending, it performs a reboot right away. It doesn't matter if these affect the patches or not. It doesn't matter if they have anything to do with patching. If there are any operations pending, you get a reboot. I think it gave a five-minute warning, with no option to postpone or cancel. That's not even long enough to call the Help Desk and get to someone that might know what processes to cancel. This applies to servers as well as workstations, of course.

I found, from looking at BindView reports that showed the pending operations, that there were many machines with pending operations from all sorts of things. Most seemed to be from the printers, others were from AntiVirus or other application updates. Some were from patches applied through baseline updates. Our results won't be the same as yours, but the results are definitely not what I had expected. I never there would be so many!

This was even worse when we were using MBSA for patching. We often had two or three separate updates for each machine, one for each scanner (MBSA, Office and Extended MBSA) with updates that month. At first we made two of the three silent, to minimize hassle for the users. That was fine until an Office update turned out to set pending operations for nearly every computer! In fact, that's what led to learning the real cause of this. Until then there were so few complaints that it never got a whole lot of attention.

What do you do about this?

It'd be great if you could control what adds to the pending operations and schedule reboots as required. That's easier said than done, based on my personal observations. We never could figure out where many of these came from.
You can schedule a reboot every night for workstations that have pending operations. If you limit this to machines with no users logged on you can avoid almost all disruption. Then you'd have to report machines that still need a reboot first thing in the morning and do something about them. If you try something like this, remember that this info is updated by hardware inventory. Be sure you're looking at current data!
If you have workstations that can't be rebooted normally, such as night operations consoles or 24-hour call centers, you'll need to report such machines and contact the users, asking them to reboot. If you have a lot of these you may need a tracking and follow-up procedure.
You could run a script that nags users to reboot until the Pending Operations registry key is clear.
You can report servers that need reboots on Friday morning and notify the appropriate teams to please reboot their servers during the weekend maintenance window. That's probably the best way to handle servers. You might do a report earlier in the week with a follow-up on Friday, to give more warning.
Think seriously about how you do baseline patching, for machines that need old patches reapplied because of application installs or maintenance activities. I'm writing a separate article to address this subject.
We were able to minimize the impact on workstations by advertising the regular patches beginning late in the evening. This did not eliminate the impact, but it came close enough to serve.

The main thing is being aware that these reports represent a real problem, and what's causing it. If you understand what is setting the pending operations you're well on the way to preventing the problem.

Baseline Patching

2007-08-07T19:49:00Z

Do you concentrate just on applying the latest updates? Do you figure that once last month's patches are done you can forget about them? If so, you are leaving some machines vulnerable without realizing it. Every time an application is installed, or uninstalled and reinstalled, there may be appropriate patches required. That's true even if they were applied previously. The same thing can happen just adding a feature to Office, if it's one that was vulnerable. If a support engineer uninstalls and reinstalls an OS component on a workstation or server, some patches that had run before may be needed again. The biggest exposure comes when you install SP2 on an XP SP1 machine, or upgrade the version of Office. These absolutely guarantee that some old updates must be reapplied, because different versions were needed for the new software versions. Other updates weren't appropriate before but are now.

What not to do
You do not need to hang on to each month's update package and rerun it every week or so. That will just add unnecessary overhead to each affected machine, and possibly disrupt operations if it causes an unexpected reboot. In this context, any reboot that's not at a scheduled time and pre-announced rates as unexpected. Even if the patch gives a 30 minute warning, that's meaningless to a manager who is away at a meeting. How about on a server outside the maintenance window, or while an engineer is performing other maintenance activities?

What you should do (the basics)
SMS 2003 provides a mechanism designed to make this easy. Create a patch deployment package using DSUW that includes all previous patches that you want to include. At my last job I think we had everything since 2002. Make sure you exclude any updates that have been superceded. Advertise it to run from the Distribution Point; do not set it to download. The individual machines will copy the executable from the DP, analyze the last scan results, and only copy locally any updates that are required. This means you can have a huge package with minimal impact on the target workstations and servers. Each month you'll add the new updates to it and remove any that became superceded. Only the changes are propagated to the various DPs, not the whole package. This even works well, under most circumstances, with remote users.

Why it's not quite that simple
You had to know it couldn't be that simple. What is? In this case the biggest catch is the reboots needed after patches are applied. If you make the patch package do reboots you will create the unexpected reboots I warned against earlier. That's true even if you make it an interactive package with a long countdown - you'll still disrupt some machines.

What happens if you just apply the updates without rebooting? That's not good either. First off, the machines remain vulnerable until they are rebooted. Since that could be a long time, this is not good. Second, this may result in another form of unexpected reboot the next time patches are applied. See my article Unexpected reboots when patching.

Suggestions
For workstations, I'd run the baseline patches silently, without reboot, once a week. I like Saturdays for minimal impact and the best opportunity for reboots. Other schedules will be better in some organizations. Make sure the scanner will always be run prior to the baseline, so that any required patches would be detected. Have a reboot package scheduled for that night, applying only to machines without a user logged on. Run reports of machines needing reboots Monday, after any daily HWI has run. Report again on Tuesday if you have many laptops, since some might be patched Monday morning. Take whatever steps are most appropriate in your organization to get them rebooted before the next patch cycle. This could be rebooting machines even if a user is logged on, writing to the user and manager, or running a script that nags them to reboot.

For servers, I'd try to give the support engineers a way to run the scanner and baseline patches after they perform maintenance that might require new patches. They could be set up so they could be run by the engineer at will through Control Panel. If that's not practical, they could inform the SMS Administrators of planned maintenance and have the scanner and patch package scheduled as part of the operation. Failing these things, or as a second level of protection, run the scanner daily and report any machines that require patches. If a server appears on that report, schedule the patching and reboot with the appropriate support team for the next maintenance window.

The most important thing to consider, always, is the business needs of the organization. Patching always involves weighing the tradeoffs between complete patching in a timely manner and minimizing user disruption. It's impossible to have both, but it is practical to find procedures that provide acceptable levels of each. The more closely you can work with the management of the server support teams, business units, and key departments like night operations and call centers, the easier it is to reach an acceptable balance.

Patch Deployment Strategy and Scheduling

2007-08-07T01:21:00Z

How do you schedule when updates will be applied to your machines? What deployment strategy and design do you use? Patching always carries the risk of unnecessary accidental disruption, but that risk can be minimized through careful scheduling, planning and processes. This is partly scheduling and partly deployment design - which groups of users can control the schedule on their machines to some degree, and which can not, and how that is performed.

How do you develop a deployment strategy and schedule? You start by understanding the business needs of your users, and the different categories of users and machines that might exist. Not only are there important differences between scheduling workstations and servers, but also groups of machines within each type.

One common strategy separates the updates from the reboots. Reboots are performed by users or scheduled independently. That leaves unbooted machines vulnerable, possibly for an extended period. Products or OS components that are partially updated, and partially waiting for the reboot, may not operate properly because of the mix of file versions in place. A better approach is to assure that the reboot happens shortly after the updates, at a time that is appropriate to the particular machine.

The general analysis process is to identify groups of machines with common scheduling characteristics, determine how to identify specific machines in each group, and then develop a deployment strategy that meets the needs of each group with the simplest possible structure. You don't want to create ten different deployment packages if you can create two or three that meet all of the needs. You also may have exception workstations or servers that are not patched through automation that you want to address in the design. The following sections address workstations and servers separately, as each group has its own unique characteristics and issues.

Workstation Updates
Here are examples of categories of machines you might have:

You probably have a very large number of workstations that are only used during day shift, and can easily be updated and rebooted at night without disruption.
Workstations used by night operations staff definitely can not be reboted at midnight without causing problems.
Call centers that operate 24-hour will be disrupted no matter when you reboot them if you do all machines at once.
You may have users, such as insurance company actuaries, who have processes that run for several days. Rebooting them during the wrong night will certainly cause problems.
You also may have laptops that are frequently away during the deployment, and remote machines that have unique scheduling issues.

It's vital to understand how you can identify the machines in any group you're considering giving special treatment. If they are in unique subnets or automatically in specific AD groups that makes iteasy. If you need to manage lists manually you must allow for the manual effort and some unknown degree of errors during a deployment. Machines will be replaced and you won't be informed. What would be the impact of such errors? Your deployment plan must minimize such potential impacts.

Companies like banks and retail chains are likely to have a large number of remote locations that can easily be identified by subnet and could easily be updated and rebooted during the night. It might be worth having one deployment process for such machines that gets them out of the way quickly, while addressing operations centers separately. That way you quickly get much of the needed protection, and the opportunity to identify and resolve patching issues with those machines, with minimal risk of disruption. Appropriate email warnings to users would take care of any individuals working late that night. Such machines will commonly have the same software configuration, or very few variations, so it's easy to test a sample group ahead of time to assure there are no conflicts.

The remaining machines have many different configurations and operating schedules. The deployment strategy I personally prefer allows users to decide when to apply updates and reboots within limits determined by security needs. This is easily done using the provisions built into the SMS Distribute Software Updates Wizard, as described in my article Settings for Distribute Software Updates Wizard. This allows the staff of a call center to distribute the reboots to avoid disruption, and machines used for night operations to be updated during idle times of day. Many other designs can work, though - the main thing is understanding the business needs and environment.

Any deployment strategy must also allow for handling emergency patches that must be applied in a very short time. Identify which categories of machines will be updated as groups under such circumstances, and assure that you can be prepared to implement this will very little notice.

Server Updates
With servers, you first need to limit all updates to occur within the maintenance windows. You also need to make sure which servers are rebooted at the same time. Rebooting servers requires planning similar to the workstations. Examples of categories you might have include:

Most servers probably can be updated and rebooted at the appropriate time without concern. The applications and services running on them will dependably shut down and restart properly.
Others are not as dependable, or the OS itself may be known to be unreliable on some machines. Any servers that can't be rebooted safely need to be identified and projects started to correct the problems. Until they are corrected, such machines require manual intervention when patching and rebooting.
Some servers might require running special scripts at shutdown or startup. Scripts might be created to allow previously unreliable machines to be rebooted safely if the scripts are run as part of the patching process.
Rebooting both machines in a cluster at the same time will cause problems. Note that SMS must be able to address each machine individually.
Other load balancing systems require similar treatment.
Rebooting all domain controllers at once is probably not a good idea either.
Rebooting the SMS servers while other servers are copying patch updates from them would also cause problems.
Other groups of servers that must be done in stages might be identified by your server support teams.
If you use Virtual Machine servers hosted on a server running a Microsoft OS, you need to coordinate updating and rebooting the guest and host systems to avoid double disruptions.
You may need or want to warn server support staff that are working on a server when updates are going to be applied and the machine reboted. This could be done through separate scripts or using interactive updates like for workstations, but with different countdown settings and time constraints. The staff could be advised how to prevent updates from running while they are performing critical emergency maintenance.

Servers in your DMZ present a very special case. These usually are the first to be patched, after necessary testing is completed, because they are the most vulnerable. Security issues mean these machines can't be part of the normal SMS configuration, so they may be patched manually or through some other process.

Most of the various reboot scenarios can probably be handled by some combination of silent updates with reboots, silent updates without reboots or interactive updates that would be managed by one of the server support team members. You might be able to simplify the SMS setup by creating a DSUW package that updates without rebooting, and using a separate SMS package to perform the reboots. That risks conflict with the emergency maintenance scenario, however, if updates are installed while an administrator is trying to reinstall some component.

Scheduling requires creating one collection for each time period. If several reboot scenarios are used you may need sub-collections or separate collections by both schedule and package type.

An easy way to manage this would be creating a new SQL table that has machine name, a management group name, a one-byte schedule code representing when the updates would occur, and a reboot code indicating automatic reboot, no reboot, interactive, or run a script. The management group would be used for management reporting. All machines that require coordinated scheduling, such as a load balancing group or the domain controllers, would have one group name. Each such group of machines would have its own group. If the scheduling is managed by other teams, the leading portion of the group name could identify the team.

You'd then create an SMS report that summarizes the update scheduling by group, linked to a detail report that lists each member of the selected group with its specific values. A report must also show all servers that don't have data in this table or have invalid data. You might create a web update form that allowed authorized people to update this data. That should also record the ID of who last updated a record, with the date and time, in the SQL data. That would be displayed in the detail report.

Baseline (Catchup) Patching
Assuring that all machines are fully patched even after applications or OS components are installed or reinstalled is a critical part of the strategy. Without that, every time an application such as an Office component is installed, or a server administrator uninstalls and reinstalls a failing software component, you might be creating a vulnerability.

This has been addressed in a separate article, even though it must be part of the overall strategy, because of the complexity of the issues this raises. See Baseline Patching.

Verifying Authorized Times and Setting Grace Period

2007-07-22T03:23:00Z

In my earlier article, Settings for Distribute Software Updates Wizard, I said that we changed the grace period on the command line as required to change the patching deadline for various deployments rather than modifying the Date Authorized.

I have created a spreadsheet VBA script to assist with this. It works with all scanners - ITMU, Office, MBSA, and Extended MBSA. First it lists all updates in the deployment package with their individual Date/Time Authorized settings. This allows you to make certain they are consistent. After that list it creates two lines where the desired deadline can be entered and the required grace period is calculated. This goes on the SMS program command line in the g:<hours> switch.

To create a spreadsheet for your use, download file ParseXML macro.txt by right clicking on this link and choosing Save Target As. Open a blank spreadsheet and paste that into the macro code window. This text assumes that you have MSXML 4.0 installed on your computer. If not, edit the line "Set xmlDoc = CreateObject("MSXML2.DOMDocument.4.0")" accordingly. While in the code window, open menu item Tools, References and check the consistent library, such as "Microsoft XML, v4.0" and click OK. Save the spreadsheet with the desired name.

To use this:

Run macro ParseXML in this spreadsheet
A "File Open" window will open.
Navigate to the folder on your SMS server where the DSUW packages are created.
Select the folder containing the desired package, and open the PatchAuthorize.xml file. Note, only xml files will be displayed.

The spreadsheet will be populated with the package name, the list of updates with associated product name and Authorized Date/Time. The cell after the date will say Local or GMT, as appropriate. Underneath the list are cells labeled Deadline and Hours. The Deadline cell is filled with the current time. Change that to the desired deadline date and time. If you only enter a date, it is treated as midnight of that date.

Note that you can create multiple programs under the one SMS package created by DSUW, with different command lines for each. This allows you to have separate programs and advertisements for each deployment, further reducing the risk of error.

If you find any errors in this program, please let me know. At some time in the future I will update this to analyze the command lines of each program in that package and list the settings in the spreadsheet. This will complete the ability to verify all DSUW settings without rerunning the wizard and risking making accidental changes.

Patching Standards or Objectives

2007-07-20T00:00:00Z

When you deploy the monthly security updates, what success rate is your goal? Sure, we'd all like to patch all of the machines every month, but that usually isn't practical. As in many areas, the last one percent can cost as much money and resources as the first 99 percent. The last one tenth of one percent might cost as much as the first 99.9 percent. What standard should be "good enough" for your organization? You should try to make an objective analysis and decision about this. That will help you decide how to handle the real-world variations we face each month.

Risk Levels
The first consideration must be the risk levels associated with different updates and categories of devices. If there is an update for a vulnerability you can't protect against through other means, like the firewall, and it could do serious damage, your standard for that particular update might well be 99% or higher. That's especially true if there are exploits in existence. You want to get that behind you, and add the update to the build process to protect in the future.

Systems in your DMZ have far greater exposure to risk than other devices, so the standard for those probably should be 100%. They should be patched at the first opportunity, after testing the patches on similar machines in the lab to assure they won't break normal operations.

Your organization may have other machines that are so critical that nothing less than 100% is acceptable. In some organizations that might include all servers. In others it might be the DCs, Exchange, and database servers. You might also have workstations that are so critical you don't want to take any risk with them. The wire transfer department in a bank comes to mind, along with workstations that perform high-value financial operations in any organization. What could happen if a hacker got control of a workstation in your Call Center? What confidential information could they get at? What about the computers belonging to senior executives?

Laptops are a special category unto themselves. They are commonly exposed far more than desktop machines inside your firewall, and patching them can be a major challenge. Yet, in many companies, they are the primary source of infections getting into the corporate network.

Resources
What resources do you have available for patching? What else do you need them for? Only the largest companies normally will have any staff dedicated to patching. More often they also perform application deployments and other activities. What resources can you take, at what cost or impact, in case of an emergency? This must be thought through in great detail so managers can make informed decisions on the trade-offs.

The trade-offs are likely to vary in different months. One month you may only be 80% patched because of an issue with an Office patch, so there's a high value to added time spent on patching. Another month may have a co-worker out sick, so the value of time helping with deployments is greater. In that case you might return to patching after the co-worker returns.

There might be staff in other teams or departments that could be borrowed to help with an emergency that required 99% patching in a short time. They must be people that already have the necessary security permissions and familiarity with the tools they would use, such as psexec.

You should have a feel for how many of the normal application deployment staff are needed for installations that can't be delayed, even for emergency patching. If you take some of the others, what will be needed to get caught up with their normal duties? If they'll need to work overtime, will you compensate them in some way? If you don't, they will resent patching if that occurs more than extremely rarely.

Automation
What are the opportunities for using automation, new products, etc to improve how many machines can be patched without manual effort?

If you use SMS, the biggest part of this is probably client health. Anything you can do to minimize client issues will help patching success rates. Just detecting and handling issues throughout the month instead of during patch deployment can make a huge difference in what can be accomplished with the available resources. Scripts available in myITforum and Dudeworks can help with this.

If you have a significant number of machines that can't be patched with your automated tools, you should have an active project to resolve whatever issues exist. If these are machines where the users just say no, see my blog Patching "exception" workstations for more detail on this subject. If the problem is the OS or the applications running on the machine, someone should be responsible for rebuilding the machine, replacing the applications, or otherwise resolving the issue. Senior IT management should be made aware of such machines, the plans for eliminating the issues, and the risks associated with any that won't be resolved within the near future.

Remote machines including traveling laptops, home offices, and small offices may present other issues. There are many possible solutions such as logon scripts, products such as 1e Nomad, etc. Sometimes the solution might be just developing processes that minimize the manual effort required. As an example, consider a company with a small remote office served by a WAN link that can't handle patch updates during working hours. You could create a scheduled task that will copy the patch files to one machine after hours, then a task on each PC there to run the updates the following night, followed by a reboot, after the file copies were verified. That would be far more efficient than patching each machine individually after hours using psexec or some other method.

The best way to attack this is to start by analyzing the machines that aren't successfully patched through your automated tools, and developing categories. Then identify the categories that can be solved most easily and work on them. At the same time, get appropriate people working, or at least thinking, about solutions to the other categories.

Communication
It's vital to make sure that the appropriate IT and business unit management, and the security team, understand the standards. You never want to face machines being compromised and someone saying, legitimately, "but I thought you were patching all of the computers." This needs to be a deliberate decision, in consultation with appropriate other people.

Summary
Overall, this is a cost-benefit and risk management analysis much like managers do all the time. The real dangers come from having to make rush decisions because things weren't considered in advance, or from having to explain why 2 percent of the workstations were attacked successfully because they weren't patched. Proper planning and communication can minimize those risks.