Steve Pruitt at myITforum.com

ConfigMgr OS Deployment

spruitt — Sun, 13 Jan 2008 18:32:00 GMT

As I promised earlier in the mssms mailing list, I just created a set of Wiki pages to begin documenting the ConfigMgr OS Deployment process. This includes a general introduction page with sections for various areas of interest plus a detailed description of the process and procedures I developed in recent weeks for creating a reference image. There are many sections that have not been completed. I hope other community members will jump in for those.

When reading these pages, keep in mind that they were developed from my initial experimentation in a test lab. I fully expect that they contain errors and misunderstandings, that I hope community members will point out and correct. Also, the security aspects are based on a lab environment that no one else could access. In real world use you should create and reference suitable accounts for image capture and generally assure proper access controls.

The one known problem with the detailed instructions is that software updates do not apply successfully before the image is captured. I'll update the instructions once that is resolved.

Desired Configuration Management - using Configuration packs

spruitt — Sat, 29 Dec 2007 01:43:00 GMT

Microsoft and several vendors have created a variety of DCM configurations, or rule sets, checking for various security requirements. These include basic checks for Windows 2003 and SQL 2005, and requirements for SOX, HIPPA, and other sets of regulations. These can provide a great basis for checking the status of your organization's environment without the need to completely create the appropriate definitions. You can see the complete catalog of configuration packs here.

The following comments and suggestions are based on my testing of a few packs created by Microsoft. Most of the comments probably apply to packs from other vendors as well.

The Microsoft packs are downloaded as MSI files. When installed they create an entry in Add/Remove Programs and a folder in Program Files containing one cab file. After that cab file is imported through the Configuration Manager console the entry in Add/Remove Programs is not needed and can be removed. I have no idea why they don't just let you download a cab file directly.

To import the definitions:

In the Configuration Manager console, right click on Configuration Baselines and select Import Configuration Data
Click Add and navigate to the desired cab file
Click Run, then Click Next
The Summary will show what Baseline and Configuration Items will be added
This is the first time you find out if the pack contained a Baseline as well as one or more Configuration Items - many only contain CIs
Click Next, then Close to complete the import operation
If the pack did not contain a Baseline you will need to create one so the rules can be used. The process is sufficiently simple that I won't go into it here.

The next step is to examine the properties of the Configuration Items to see what is being tested, then advertise the new Baseline to a small representative sample of computers to see the results. In my testing, two of the Microsoft Configuration Items tested for file versions that were already upgraded. The tests required an equal match, so errors were reported for machines that had newer files. If you run into this, see my previous blog article for instructions for editing the incorrect CIs. Think carefully about what file versions you want to test for, to properly reflect the desired service pack and update levels.

The standard Configuration Manager DCM reports include many compliance reports. As with most such reports, you can get a high-level summary report and drill down to details of which exact validation tests failed for each computer. Analyze these reports carefully and decide on a plan for correcting the deficiencies reported. You can also see detailed compliance reports for any computer through it's Control Panel Configuration Manager applet by selecting the desired report in the Configurations tab.

In many cases the remediation will require applying a service pack or selected updates to all applicable computers. Those can be handled efficiently through Software Updates. If some errors are reported for just a portion of the applicable machines, but enough of them to warrant automated solutions, you can create collections based on the DCM results and distribute applications or scripts that will apply the required changes. The WQL queries you can create can not reflect the lowest level of detail, of specific validation tests. The finest detail is represented by the list of rules shown in the Configuration Manager console under each Configuration Item. These rules often contain one or very few validation tests, but some may contain several dozen or more. The scripts or applications you advertise should allow for this and only make the updates required on a particular machine. In extreme cases, you may need to use the editing capabilities described in the previously-referenced article to divide the validation tests into two or more rule sets that can be addressed individually for remediation.

For more details about creating remediation collections, see How to Remediate Non-Compliant Computers Using Software Distribution. If you have problems with any of this, see Troubleshooting Desired Configuration Management Issues.

Updating Desired Configuration Management Baselines

spruitt — Fri, 28 Dec 2007 01:47:00 GMT

I've been testing the new Desired Configuration Management feature, and I see it has fantastic possibilities for helping to manage a company's infrastructure. However, I also quickly found one problem. If you use the standard Baseline Configuration Packs that can be downloaded from Microsoft, you'll find that some of the file version tests are obsolete. I ran into this for Windows 2003 DNS and Basic SQL 2005. Fortunately, we can update these definitions to reflect the service pack and update levels we want. The process is documented at TechNet, of course, but I had trouble following it at first. You can't directly edit the configuration data you download and import into DCM. If it's not created at your site it's read-only. The solution is to create duplicate versions of the components you want to edit, edit those, then substitute them in a duplicated baseline. Note: sometimes you will be downloading configuration items and creating your own baseline. In those cases, you don't need to duplicate the baseline, only the desired configuration item. Here's my step-by-step process that worked:

The first step is deciding which tests you want to change. You can use the following steps to examine each rule and research them, but I prefer the simpler way... run the selected configuration tests against a selection of typical machines, then analyze the errors that are reported. Some will really be things you want to correct, others will reflect obsolete tests that need to be updated.

Here's the steps to edit the baseline and validation rules:

Expand the list of configuration baselines
Right click on the baseline you want to edit and choose Duplicate
Enter a new name for the local copy of the baseline and click OK to complete creating a duplicate copy
Expand the Configuration Items section and select the rule you want to edit in the right hand pane
Right click on the rule and select Duplicate
Enter a new name for the local copy of the rule and click OK to complete creating a duplicate copy
Select the new copy of the rule, right click on it, and choose Properties
Select the Objects tab
Double click the object you want to edit, then select the Validation tab
Select the desired test (there's usually just one) and click Edit
Revise the test as desired. In my example, I changed it from 'Equals' to 'Greater than or equal to'
Click OK to close window
Repeat as needed for other objects in this rule
Click OK to close remaining windows
Select Configuration Baselines in the SCCM Console tree
Right click on the desired baseline (local copy if you made a duplicate earlier) and choose Properties
Select the Rules tab
Click the appropriate underlined configuration item category, such as "applications and general" as shown above to display a list of all available rules of that type
Select the newly edited rule to add
Click OK
In the Rules window, select original version of rule, click Delete
click OK to close the window
If you duplicated the downloaded baseline, now you want to make sure you don't run it any more. You can do this in several ways:

Remove the association with a collection:

Open the properties of the old baseline
Select the Assignments tab
Select the collection(s) and click Delete

Disable the baseline by right clicking on it and choosing Disable
Delete the baseline by right clicking on it and choosing Delete

If you replaced the original baseline, remember to assign the new baseline to the desired collection(s).

Test your changes on a small pilot group before assigning it to a broad collection, of course. If you used a pilot test to discover the original problems, the same collection is ideal for testing the changes.

Deploying applications as software updates

spruitt — Wed, 26 Dec 2007 15:23:00 GMT

Have you ever needed to decide how to deploy an application that requires a reboot? Has a manager ever asked if you can deploy it as a software update, using the same notification and scheduling options? I've faced both, and I'm sure most SMS admins have as well.

SMS doesn't lend itself to deploying applications that require a reboot. There are no notification and scheduling options available, as there are for updates, and the update process is designed just for security updates from Microsoft. You need to either set up the program to only run when no user is logged on or wrap the installation in a custom script that provides the needed capabilities.

This can be much easier under Configuration Manager, if you also install System Center Updates Publisher (SCUP). Updates Publisher extends the software updates process to support update catalogs from other vendors and also custom updates created by the SMS Administrator. You could create a custom update to deploy your application installation that requires a reboot. SCUP updates are completely integrated into Configuration Manager, and are deployed exactly like other updates using the same capabilities. The MyITForum Wiki article about SCUP provides some additional information.

The basic idea is to create a custom update rule that will install the application, and deploy it to a collection of target computers. It's not quite that simple, but close. There are a few key things you need to do.

First, prepare the installation as an exe or msi if it's not already in that form. SCUP only supports exe, msi and msp files. If you use an exe, document the return codes that indicate successful completion and those that indicate a reboot is required. Test to be certain your command line results in a successful silent install as a new install or an upgrade.

Second, you need to plan the rules you will use. An SCUP definition uses three rules (all are required):

Prerequisites: these are things that must be present (or absent) before the software can be installed. You can test for the operating system, files and versions, registry info and WMI query results.
Applicability: files, registry info or WMI queries that make the update applicable. For an application, you might use something generic that all computers would satisfy.
Installed: files, registry info or WMI query responses that indicate the application is installed. If the installation uses an msi, the new rules wizard will automatically extract the installation code and create a rule for that. If you create your own rule, be sure it will accurately reflect failed installations and if the application is uninstalled. This rule prevents the application from being reinstalled every time updates are reevaluated.

Keep in mind that after the custom update is published, it will be part of the WSUS database for all machines. All computers will scan for this being applicable, and that status will be included in software update reports. This will only be installed on computers included in collections the update is deployed to, no matter how many others report it as applicable.

Once the update is published, it is deployed exactly like any other update. You would create a deployment selecting the one update, choosing the desired notification and scheduling options and targeting the desired collection. This will require careful thought and testing to be certain everything works as expected. For example, if regular update deployments specify the option to install any other updates due within a certain time period, this might include your custom update and cause it to be installed before you expected.

Since this is deployed as an update, normal software deployment reports don't apply. Before using this capability, be sure you understand which software update reports you will use to monitor and manage this process.

The same concept might be used with SMS 2003 and ITCU (the predecessor to SCUP), but ITCU was rarely used. This is partly because ITCU was not integrated into the SMS deployment process as well as SCUP is with Configuration Manager.

Reboot before patching

spruitt — Thu, 20 Dec 2007 18:28:00 GMT

We've all experienced users that blame absolutely everything on patching. My personal favorite is the person who blamed us for the problem that started just after she received my email announcing that patches would be applied the following week.

Many of the reports are more reasonable, because the problem really did start when patches were applied. This is frequently because some other change had file operations pending the next reboot, and many users only reboot when patches are applied. There's no way the users can distinguish those results from ones really caused by patches.

If this is a significant issue for your organization, one easy solution is to schedule a reboot of all workstations a few days before patches are deployed. You should exclude any that are in a special-handling collection, of course. That should make most unrelated issues turn up before patching starts, making them far easier to diagnose. This can also reduce the problem of SMS 2003 patch deployments causing a reboot as soon as the advertisement starts, as described in my earlier article on that subject.

Configuration Manager (SCCM 2007) Software Updates

spruitt — Mon, 17 Dec 2007 01:43:00 GMT

I just finished writing the initial Wiki articles about Software Updates in Configuration Manager, and hope others will contribute to it. It covers migration from SMS 2003, installation and configuration, deployment design, reporting, suggested normal monthly activity schedule, and troubleshooting. Of particular value to anyone who has begun testing this process is the section on deployment options in the deployment design section. This lists all of the major options you can select, describes how they work, and alternate places where they can be selected.

One intended section remains to be written, about updating servers. The primary feature that affects server deployments is the ability to restrict updates to maintenance windows specified by collection. That feature is already described in the deployment design section, so I'm trying to decide what else to say about updating servers.

If anyone sees errors or thinks of other topics to be covered, feel free to dive in and edit the material - or comment to this post or write to me.

Patching in SCCM 2007

spruitt — Tue, 20 Nov 2007 22:44:00 GMT

Here’s a summary of some of the most significant differences I found in patching between SMS 2003 and SCCM 2007. Note that other differences I don’t list may be significant to your deployment process. It’s vital to thoroughly test this and plan the changes you’d make to your environment. Also, these reflect the results of my testing; your results may be different. Also, read Chris Mosby's blog, where he offers a preview of Chapter 14 of his book about SCCM. It describes the Software Update process in SCCM

ITMU becomes WSUS (plus more)

Greater variety of updates can be deployed

ITMU supports nearly all security updates
WSUS supports all updates that are in Microsoft Update
That, in turn, means you need to pay closer attention to what’s newly released. It may include non-security updates that you don’t want to deploy.
This also makes it much easier to deploy these non-security updates, if desired.

Install ITCU to be able to deploy non-Microsoft updates from hardware vendors, Adobe, etc. It’s fairly easy to add your own upgrade deployments for any applications or products.
The new Desired Configuration Management capabilities let you use patch scanner-like analyses of all sorts of characteristics; this can be used in conjunction with patching or independently

Scanning

Synchronization with Microsoft Updates happens on a specified schedule and can be run on demand
SCCM does not allow you to schedule scans separately by collection; there is one global schedule for the system
Scans are initiated by the client as part of various activities, along with the scheduled scans
Scans run at random times during the two hours after the machine policy is received, to spread out the load on the network and servers
There is no apparent way to schedule scans on specific dates, so the only way to assure accurate patch requirement reports at the beginning of a patch cycle is to run scans daily
It is easy to change the scanning schedule, so you could easily scan daily for the first days of a patch cycle and use some longer interval the rest of the month
There is a separate schedule for rescanning based on updates that were scanned previously (i.e., no attempt to find other updates to scan for)

Deployment Scheduling

You can still deploy with different options to different collections
It’s now much easier to use staged deployments:

Create an Update List of the updates you want to deploy
Create multiple Deployments targeting the desired collections with the desired options
Use Deployment Templates to assure consistency
This can include a mix of interactive and silent deployments with different schedules
Only one Deployment Package is created and distributed to the DPs

In SMS 2003 you can create a package that allows postponement for a specified time from either the Time Authorized or the Time Available
In SCCM the deadline is only based on Time Available, which is the “advertisement” start time

Baseline Patching

SMS 2003 requires that you have a separate deployment package of updates from past months with a separate advertisement as the most efficient way to rescan for past updates and reapply them as required
SCCM has a rescan schedule that is distinct from the regular scanning schedule
SCCM rescanning can use the original update deployment packages; there is no need to combine them
SCCM software updates will get update executables from any available package; that means you could combine updates from previous months into rollup packages for easier management if desired
Many other ways of combining past and/or current updates are possible, to accomodate any desired update management practices
Keep in mind the effects on network traffic and DPs of adding or moving updates between packages - make sure your plans will be effective and efficient at all levels
Also remember that the most valuable reporting may be based on the deployment structure

Migration

You have to uninstall all scanners except ITMU before upgrading from SMS 2003 to SCCM 2007
If you do a side by side upgrade you can install ITMU under SCCM 2007 to support SMS 2003 client
If you can’t complete updating all clients from SMS 2003 to SCCM 2007 between two patching cycles, keep one SMS site server operating in case you need to deploy any ESUIT updates. These can not be deployed through SCCM 2007.
Install WSUS 3.0 before SCCM, and cancel out of the wizard without configuring it –SCCM will configure WSUS when that site role is installed

User Interface

The appearance of the balloons and dialog boxes, and the steps needed to run the updates now or a specified later time are different
If an update does not require a reboot, there’s no dialog box saying the update has completed as there is with SMS 2003
The deployment process and schedule at your company may change if you take advantage of new features
All of these changes must be communicated clearly to users shortly before your first SCCM update deployment to minimize confusion and calls to the Help Desk, and the disruption resulting from such confusion
If there are file changes pending a reboot when new updates are distributed, the user is informed in the regular dialog box - the system does not force the reboot until the patching deadline (note: I’ve heard reports of other behavior, but my testing has been consistent)

Reporting

I found it frustrating not having the advertisement-based reports and queries for scanning and updating that I was used to
The new reports are based on the current patch status of each machine, not the status of the scanner and deployment executions plus patch status
Be prepared for this, and think carefully about what reporting you’ll need
There are many new reports – study them before creating your own, you may not need them after all!

Patching and Communications

spruitt — Sun, 21 Oct 2007 23:47:00 GMT

Patch Management is only partly a technical activity. Good communications are at least as important, if not even more so. No matter how well you designed your deployment plan to support the needs of your business, you need to communicate effectively with the right people each month to be successful. The general principle is to avoid surprises. If everyone knows what’s going on that concerns them, the whole process proceeds smoothly. What things should you communicate, and who do you tell? The answer always depends on details of how your company is organized, how it operates, and how you deploy the updates. Here are some examples to consider:

Anticipated Patching

What: On Thursday before Patch Tuesday, Microsoft announces the updates they expect to release the following week. Keep in mind that the actual release may contain more or fewer updates and could also include re-released updates.

When: Shortly after the pre-announcement is available.

Why: This announcement doesn’t contain a lot of detail, but it’s enough to have some idea how great the impact may be.

Who to tell: Share a summary of this announcement with everyone involved in Change Management, so they can begin thinking of possible impact on other planned activities.

Patch Release Details

What: Summary of details of each update, including the affected operating systems or products, how the vulnerabilities could be exploited (i.e., through email, web sites, network connection or physical logon), if the update requires a reboot, and if it can be uninstalled.

When: On Patch Tuesday afternoon or first thing the following morning.

Why: This helps people understand what applications might be affected and the impact of the deployment.

Who to tell: Everyone involved in Change Management or testing of server applications.

Deployment Schedule and User Impact

What: The planned schedule for deploying updates to pilot testers, other special groups, and the general population, including the deadlines for each group that is allowed to postpone or schedule updates. Explain the normal activities required and any unusual activities or appearance. Include a link to a comprehensive explanation of the patching process for users.

When: The afternoon after Patch Tuesday, or as soon as the schedule and activities are known. This should be after initial testing is completed.

Why: If users know what to expect to see and what to do, you’ll have fewer calls to the Help Desk and fewer problems. This also serves to explain the patching process to new employees. This also helps many people schedule changes or other activities to avoid conflict with the updates.

Who to tell: All employees that will see update activities. You may want separate communications to pilot testers and other users that have different schedules or activities than the majority of users. That allows simplifying the message to all employees and reduces confusion.

Special Deployment Announcements

What: These are announcements to pilot testers, users responsible for exception machines that get non-standard deployment options, and anyone else who sees deployment schedules or options that are different from the majority of users.

When: On the day updates are deployed to each group, or very shortly before.

Why: These announcements serve as a reminder of the update schedule for their particular machines, a reminder of which machines are affected, and a reminder of the testing and feedback that is expected of them. This is intended the improve the quality of pilot testing and feedback, and reduce problems resulting from updating exception machines.

Who to tell: All users who will receive deployments on a different schedule than the majority of users or who have different options for postponing or processing updates.

Deployment Issues

What: You will occasionally find issues with certain updates that change the deployment plans. These might include problems that prevent proper patching or conflicts with important business applications. The information you supply includes the affected update, the issue preventing normal deployment, the affected machines, the revised deployment plan, and the plan for resolving the issue.

When: As soon as the issues are found and the revised plan is decided.

Why: The affected people need to know if an update will be deployed separately, later, or if the entire deployment is being delayed, or if an application is affected by an update.

Who to tell: In each case, the Information Security and appropriate IT management must be informed. Other people may need to know if they have to take different actions because of these changes. Any more general announcements must be written and timed to minimize confusion that can result.

Patching Progress Reports

What: Management reports of the progress applying updates and explanation of any differences from normal.

When: The management receiving these reports will specify the desired frequency. In larger organizations there may be separate detail reporting to lower levels of management and summary reporting to senior management, with different frequencies. Typically there are a small number of regular reports at key dates, plus exception reports if required.

Why: IT and Security management will generally want progress reports for any important projects. Security patching, even though it occurs every month, qualifies in most companies. In addition, if problems arise in any portion of the deployment it’s vital to inform the proper people as soon as possible. For example, an update to a particular product may frequently fail for some reason that didn’t turn up in initial testing. This would require developing a plan for rerunning that update, which may entail unplanned user disruption.

Who to tell: This normally includes the two levels of management above the person managing the deployments plus the Information Security manager. In some organizations, management of server support, desktop support, division CIOs, and others may want to be included in such reporting.

You have to tailor this to your organization, of course. The key is to remember that patching affects every computer, every user, in the company. If the users and managers understand what will happen and when, everything is likely to go much smoother.

Patch Process Design and Communications

spruitt — Sun, 21 Oct 2007 23:32:00 GMT

Have you ever designed a patching process, only to find that users and managers complained after you implemented it, and no one supported you? Good planning and communications with the right people help assure that you develop a good plan, that the plan is approved by everyone concerned, and minimize problems and complaints during the deployments.

Before designing a patch deployment methodology you must understand your tools and objectives:

Understand the options available to manage the deployment. These are different for WSUS, SMS 2003, ConfigMgr (also known as SCCM 2007) and other software. If you use SMS, you can combine the patch deployment options with other SMS capabilities such as running only if no user is logged on, wrapping the deployment execution in a script that adds options, etc.
Get your management’s guidance and expectations. What are your trade-offs between security and disruption? What standards are expected for completing the patching? The best security means completing the patching quickly, but that also requires the greatest disruption. Avoiding disruption extends the time needed to complete securing the network. This should be one of your first questions to your management. The preferred guideline is in the form of “Use the least disruption that allows completing patching to 99% within ___days” or something similar.
Understand the business needs of your environment. That means talking with key business unit and IT management.

Your plan has to include a few key parts that you can develop through communications:

What are the restrictions on rebooting computers in various groups? Some can be scheduled, some require control by the computer users or other responsible people.

Most companies have some groups that can be rebooted any night without problems, others that can never be rebooted during the night, and others that need to control the timing. For example, if you have a 24-hour call center, you can't do anything that will reboot all of those representatives at the same time. It needs to be spread out and controlled. Operations consoles can't be rebooted during their working hours, which are often the middle of the night. There are usually some IT managers, such as Desktop Support or Call Center, who can identify various such groups and the managers there to talk with.
Will reboots occur immediately after the updates are applied or some time later? Will they be automatic or manual? If manual, how will you manage and enforce the requirement?
How much time do various group need to schedule reboots? For example, Insurance company actuaries often run calculations that require several days to complete. Do you have any similar functions?
How will you handle computers, such as laptops, that are offline during the normal patching period? Will they be patched immediately after they reconnect, or will users be allowed a period to complete this?

How will pilot testers be selected, and replaced as required? Who will test for a department if a pilot tester is on vacation?
How will you identify the computers that fall into different groups, and how will you maintain the accuracy of those lists? That includes groups that are treated differently for deployment schedules or options and pilot testers. Maintaining the accuracy of these lists is vital - errors in these lists will cause the biggest problems and complaints most of the time. If possible, find a way to identify critical groups of computers automatically by subnet or some other common characteristics that SMS can use in collection queries.
Who do you need to communicate with on a regular basis? How will those lists be maintained? These may include users that will get different deployment handling than most users, pilot testers, managers of key departments, etc.

Find out which departments are particularly critical to the business, and talk to them to find out if they have any special needs. Find out which departments are known to complain to IT senior management, and include them in your planning from the beginning. You need to come up with a patch management system that will meet the needs of all of these groups. Talk to each group and find out their requirements. Explain the importance of protecting their computers with the minimum of disruption.

Once you develop a plan that you think will meet their needs, you have to convince a number of groups of people that your plan is sound. That means selling it, and also listening closely to all complaints or objections. In most cases you should be able to explain how the issues are handled in the plan. Sometimes you'll need to think about the issues the raise and revise the plan accordingly.

The first group is your management. You can't go any further until your boss and perhaps one level higher have agreed to support your plan.
The second group is the corporate security manager or team. If they believe you have a good plan that adequately protects the company, they can help convince other people.
The third group is all of the other managers you talked with while developing your plan. Show them how your design reflects their needs, and explain their responsibilities to assure that their computers are protected with the minimum of disruption.

Finally, if you are changing what the users will see or what they must do, you have to explain this to all of them. Use all of the communications opportunities your company offers, including newsletters, regular meetings with all managers or all employees, etc. Follow this up with an email to all employees that has a brief explanation and a link to a more complete presentation for those that are interested. Ask your corporate communications staff to help prepare and send this out.

If you follow this communications process, you should have the understanding and support of the IT and business unit management that are most affected by your patching strategy. This will go a long way towards minimizing problems as you implement the plan.

Recurring advertisements are required for patching

spruitt — Tue, 16 Oct 2007 17:31:00 GMT

My co-worker Shaun Cassells made an important discovery when we began patching with SMS 2003. When you advertise the patch update packages, you must always set up a recurring execution. The logic of the patchinstall process requires this if users are permitted to postpone the updates. See KB842717. If you have some update packages that allow postponement and others that don't, you're best off being consistent and using recurring advertisements for all of them. That greatly reduces the chance of creating this error. If you have DSUW create the advertisements it will always create a recurring schedule.

If you have an update that you really only want to run once, such as for pilot testers, just select a monthly recurrence schedule and delete the advertisement before the month is up.

In general, it doesn't hurt to have updates run on a recurring schedule. The update will rerun the scanner, and then only apply any updates included in the package that the scanner found were applicable. Once a set of updates are applied, rerunning that package will do nothing unless changes to the computer made one of them applicable again.

Scanners should also have a recurring schedule that assures they will always be run before the update package. Although the update package repeats some scanning functions, that does not replace running the full regular scanner. The regular scanner detects updates that are applicable or installed. The scan function within the update verifies if that information is still accurate, but won't perform the initial detection.

Using psexec to install the SMS client

spruitt — Mon, 01 Oct 2007 19:23:00 GMT

Sometimes the easiest way to install the client may be running it with psexec. That's especially true if the machine hasn't been discovered by some mechanism. The options for doing this often cause confusion, because there are two files involved - ccmsetup.exe and client.msi. CCMSETUP looks for client.msi in whatever folder ccmsetup is run from. If you use the -c switch that's normal with psexec, to copy the file being executed, it only copies the exe. That then hangs when it can't locate the msi, and the ccmsetup.log file is filled with messages reporting that it can't find client.msi in C:\Windows\System32 and will retry in 20 minutes.

The solution is to run ccmsetup from a location containing both files. The easiest is usually on an SMS site server. The proper command line is:

psexec -s \\computer \\SMSserver\client\i386\ccmsetup.exe [ccmsetup switches]

The -s command says to run using the system account. Do not use -c, as that will cause the errors described above. Using -d is optional; ccmsetup terminates pretty quickly. I generally leave it off so I can see the zero return code.

Patch Management Process Overview

spruitt — Fri, 28 Sep 2007 19:02:00 GMT

I've seen several postings recently from people just getting into SMS patching that don't know where to start. This article is intended to help them understand the basics of the process, and link to various web pages that either provide greater documentation or are used in normal operations. This is based on SMS 2003 with ITMU.

Patch Management Principles
These articles explain the overall concepts and principles involved:

Getting Started
First, install ITMU on your Central Site Server. You may also need the Extended Security Update Inventory Tool, which supports products that ITMU does not support. That can always be installed later if needed. Then go through a small test of the normal monthly cycle, described below. That will help you understand what all the articles are talking about. Then you're ready to begin reviewing the peculiar needs of your business, and design a patching strategy that fits. Ask questions in the mssms mailing list when you need help.

Monthly Cycle
Microsoft gives a heads-up of what they expect to release on the Thursday before Patch Tuesday at their Security Bulletin Advance Notification web page. Patches are normally released on the second Tuesday of each month. Their Current Security Bulletins web page links to the current month summary, which in turn links to a detail page for each individual bulletin. That's usually available beginning about noon Pacific time, but that's not guaranteed. It takes a while for all of Microsoft's web servers to be updated, so you'll often find that some links don't work until later in the day. When you are getting started, you should study the bulletin summaries for a couple of recent months, and the pages for each individual bulletin, in detail. Each type of page has sections with a + in front of a header line. Click on that to expand the section. When getting started, you should expand and read every section of every page for the previous couple of months. At the end of that time you'll have some idea of what information is available where.

You should also read the current month Detection and Deployment Guidance article. The articles for each month are linked from KB910723. These tell you which updates for which products can be detected and patched through ITMU, and which require the Extended Security Update Inventory Tool (ESUIT) or other means.

Sometime during the evening of Patch Tuesday you should run the ITMU Synch program. This downloads the latest updates to your server. Schedule the ITMU scanner to run on machines later that night, so you'll know which updates are needed in your environment. There are two scanner programs. The expedited version runs hardware inventory after the scanner completes. That sends the scanner results to your server for review, and causes some amount of network traffic and server activity. If you have a reasonably large network, you might use the expedited scanner on your pilot test machines and the regular scanner on the others. That way you'll have pretty good information available first thing Wednesday morning, and better counts by the end of the day. If the ESUIT scanner is needed for your environment, the latest version is downloaded manually from the ESUIT download page. That's installed on your server, and the scanner runs just like the ITMU scanner.

On Wednesday, you review the new updates that apply to your computers. This is shown in the Security Updates section of the SMS console. Sort by the bulletin number column to make it easy to find the latest updates. Then decide how to deploy them. This allows for the risks the updates protect you from and the risks inherent in the updates. You should always test all updates on a set of pilot test computers before the general deployment, to see if they conflict with any business-critical applications. Read my Patch Testing article for more details. You can also get valuable information from the Microsoft Security Response Center Blog and the Patch Management mailing list. Remember that reports there won't always mean problems in your network, but they will give you advance warning of possible issues. Some reports in that mailing list are over-reactions, just as some of your users will blame any problem on the patches.

Then you'll create one or more patch deployment packages using the Distribute Software Updates Wizard in the SMS console, and run them on test machines. I always included my own PC as one of the first test machines. If everything looks good, run them on all machines.

The other articles in my blog explain concepts for setting up your deployment and testing strategy. They'll make the most sense to you after studying the updates from the last few months at Microsoft's web site, and working out how you'd deploy them through SMS.

Again, ask questions in the mssms mailing list when you need help. Patching is not simple, and it's an area where it's critical to be careful. Remember, when you are updating every computer in your company, there's no such thing as a small mistake. Any error can affect the security of your company or the degree of disruption you cause your users.

Allowing selected users to postpone patching

spruitt — Sun, 26 Aug 2007 19:52:00 GMT

Someone asked an interesting question in the mssms mailing list recently. I answered it there, and thought it was worth recording in a blog as well. He wanted to know how to deploy patches so that some users could postpone the update, while the others could not. It's actually fairly simple. What follows is even easier than my original email response.

First, create an interactive patch package with the Date Authorized in the past, and allowing the maximum postponement you want for the group that's allowed to postpone the patches. See Settings for Distribute Software Updates Wizard for details.

Second, create a second program in that SMS package that's identical to the first. Use a name includes something like "No Delay" and change the g: parameter in the command line to G:1. That allows one hour postponement from the date authorized. Since that time is long since past, this program will cause patches to be run as soon as the countdown time expires, then reboot after a second countdown.

The third step could be the hardest. You need to get the collections set up properly. First create a collection of computers where you want to allow postponement, based on whatever user-to-machine data is available to you. Second create a collection using a query that specifies all machines not in the "allow postponement" collection, and limited to your existing production deployment collection. That will end up being all machines you want to patch except those that can postpone.

Finally create the advertisements. The "Allow postponement" collection is advertised to the SMS program that was created by the wizard, and the "All other machines" collection is advertised to the "No Delay" program you created manually.

In another email in that thread, Marcus Oh suggested another solution that would be better in some environments. His idea was to create a program that checks which groups the logged-on user is in. If the user is a member of Group A, the program would then execute patchinstall.exe with the command line options to postpone, otherwise run it with the command line options no not postpone. You'd update the SMS program command line to run this program instead of patchinstall.

As always, test carefully before proceeding!

Identifying and selecting Office patches

spruitt — Sun, 19 Aug 2007 02:09:00 GMT

If you are preparing your first Office patches, you may have a hard time finding them in the Distribute Software Updates Wizard. Office patches are set up a little differently than OS patches. How differently depends on whether you are using ITMU or the SUSFP Office scanner. In each case, the main problem is that the patch for each affected product has it's own Q number. These are listed in the bulletin web page, but I prefer to identify them through SMS. That's especially valuable for updates such as XML that may have both OS and Office components affected.

ITMU
The SMS console's Software Updates section has all updates and can be sorted by any of the columns. In ITMU, Office patches have the Q and bulletin numbers in the respective columns. Click the heading of the Bulletin number column to sort by that data, then locate the current updates. You'll find all of the product patches together. I like to export this list to an Excel spreadsheet and sort there, then delete all rows I'm not concerned with for that deployment. That makes it easiest to make sure I locate and select all of the proper patches in DSUW.

The individual patches can be located in DSUW the same as for OS updates, by entering the Q number as the appropriate filter.

SUSFP Office Scanner
If you are using the older Office scanner, either because you haven't upgraded to ITMU or because you still have some computers with Office 2000, it's trickier. The Office patches are listed in the SMS console's Software Updates section without any data in the Q or bulletin number columns. The first step is to export the Software Updates data to a spreadsheet. Then use Find to search for each of the Q numbers listed in the bulletin web page and move all matching rows to a separate sheet. Watch carefully for Q numbers that appear on more than one line, and move each of them.

These patches are located in DSUW by entering the Q number as the filter for the Name field. Make sure you locate and select all of the appropriate patches for all products in your environment.

General
As you search for these updates in the spreadsheets or console Software Updates, make certain you identify all patches you need to deploy. In some cases they will be listed under different scanners. They might have updates for OS components, Office products supported by ITMU, and Office products that are not supported by ITMU. Be sure to make a complete list of the patches to be selected in DSUW. Always be on the lookout for variations and exceptions. Check the counts reported in Software Updates after your scanners have run on most machines and the results reported through hardware inventory. Be prepared for surprises.

I have also seen that computers with a mix of Office versions installed, such as Office XP with a different version or SP level of Front Page, Access or Power Point can mess up the scanners. That seems especially true of products from the same version but different service pack levels. That can easily happen if Office products that may be installed separately, such as Front Page, are not carefully kept at the latest service pack level, including updating the installation packages. Baseline patching can help with this, especially if it includes the proper Office service packs.

Unexpected reboots when patching

spruitt — Tue, 07 Aug 2007 20:19:00 GMT

Have you ever had users complain that their computer rebooted unexpectedly about the time you deployed patches? Maybe it gave them a brief warning, but no chance to postpone? We ran into that, and it took a fair bit of digging to get the answer.

We found out that one of the first things that the patch update program does, even before looking at the options you selected, is to check the PendingFileRenameOperations registry key. If there are any operations pending, it performs a reboot right away. It doesn't matter if these affect the patches or not. It doesn't matter if they have anything to do with patching. If there are any operations pending, you get a reboot. I think it gave a five-minute warning, with no option to postpone or cancel. That's not even long enough to call the Help Desk and get to someone that might know what processes to cancel. This applies to servers as well as workstations, of course.

I found, from looking at BindView reports that showed the pending operations, that there were many machines with pending operations from all sorts of things. Most seemed to be from the printers, others were from AntiVirus or other application updates. Some were from patches applied through baseline updates. Our results won't be the same as yours, but the results are definitely not what I had expected. I never there would be so many!

This was even worse when we were using MBSA for patching. We often had two or three separate updates for each machine, one for each scanner (MBSA, Office and Extended MBSA) with updates that month. At first we made two of the three silent, to minimize hassle for the users. That was fine until an Office update turned out to set pending operations for nearly every computer! In fact, that's what led to learning the real cause of this. Until then there were so few complaints that it never got a whole lot of attention.

What do you do about this?

It'd be great if you could control what adds to the pending operations and schedule reboots as required. That's easier said than done, based on my personal observations. We never could figure out where many of these came from.
You can schedule a reboot every night for workstations that have pending operations. If you limit this to machines with no users logged on you can avoid almost all disruption. Then you'd have to report machines that still need a reboot first thing in the morning and do something about them. If you try something like this, remember that this info is updated by hardware inventory. Be sure you're looking at current data!
If you have workstations that can't be rebooted normally, such as night operations consoles or 24-hour call centers, you'll need to report such machines and contact the users, asking them to reboot. If you have a lot of these you may need a tracking and follow-up procedure.
You could run a script that nags users to reboot until the Pending Operations registry key is clear.
You can report servers that need reboots on Friday morning and notify the appropriate teams to please reboot their servers during the weekend maintenance window. That's probably the best way to handle servers. You might do a report earlier in the week with a follow-up on Friday, to give more warning.
Think seriously about how you do baseline patching, for machines that need old patches reapplied because of application installs or maintenance activities. I'm writing a separate article to address this subject.
We were able to minimize the impact on workstations by advertising the regular patches beginning late in the evening. This did not eliminate the impact, but it came close enough to serve.

The main thing is being aware that these reports represent a real problem, and what's causing it. If you understand what is setting the pending operations you're well on the way to preventing the problem.