As I mentioned in a previous blog post, we recently switched our monthly patch management function via SCCM away from the built-in Software Update Management (SUM) feature to the Task Sequence Engine.
One of the primary drivers behind making the technology shift was the increasing frequency at which we were deploying software (applications, non-security updates, etc.) inside the same maintenance windows used for software updates.
Without the task sequence engine there is no easy way to coordinate when the software installs and patching would happen; we had started scheduling software during the second half of the two hour maintenance windows but as the distinct software distributions started stacking up keeping up with the install schedules became increasingly difficult.
The other benefits of this change should become clear after I explain the steps/groups inside of the task sequence.
Here is a simplified example of what our monthly patch management task sequence looks like:
Inject Random Sleep: A simple random sleep timer to spread out the execution of the task sequence.
Pre-Patch Custom Actions: This step runs any custom actions to make a server ready for patching, stopping services, failing over resources, etc.. These custom scripts are provided by the machine owners and assigned to the machine via collection variables. This is one of the only steps where we do not set the “Continue on Error” option so that if a machine fails to be made ready for patching we have an opportunity to stop processing further steps.
Install Required Software Updates: The heart of the task sequence; this step installs all of the mandatory security updates targeted at a machine. Prior to patching the machine will also be rebooted if any pending reboot operations are detected. Following the patch installs we run our Update Install Source script to gather data about “who” installed the security updates (see this previous blog post for more information).
Install Required Non-Security Updates: This step allows us to install any QFE’s/Hotfixes that are required. Since we would not want to reboot after each update we declare a variable to detect if a reboot was required.
Install Required Software Distributions: Install or modify any required software distributions that are relevant to the machine.
Agent Maintenance: This step allows us to perform upgrades or maintenance on our core agents, SCCM, WindowsUpdate, OpsMgr, FEP.
Post-Patch Custom Actions: The clean-up phase of the task sequence.; this step runs any server owner provided custom action scripts to make a server ready for production again post patching, starting services, reclaiming resources, monthly maintenance steps (log archiving for example). We also handle any pending reboot operations that may have been caused by agent maintenance, software distributions, or non-security updates installed during the task sequence execution.
Switching to patching via the task sequence is not without its challenges; it requires some major changes in the way that you target machines, configure update policies and maintenance windows.
Here are some key lessons learned from our implementation (so far):
If you use the SCCM native SUM feature you can get by on collection level maintenance windows alone to control when updates are installed. I found out in testing that you can’t control the execution order of generic task sequences and SUM deployments; only OSD task sequence types get a special maintenance window override flag. When we tried to use collection maintenance windows to advertise the task sequence the SUM feature would start separately at the same time and then reboot the machine causing the task sequence to fail. The SCCM advertisements will deliver the task sequence and also provide the bounds for the the maintenance window, using a combination of Mandatory Time and Expire Time, instead of the collection level maintenance windows.
Past Deadline SUM Deployment Policy
We need agents to install, not just be offered, any mandatory updates during the maintenance window. Also, machines will only pre-download the required update files before the maintenance window if the policy is past deadline. The policy is configured to only run during maintenance windows and prevent system reboots outside of maintenance windows.
Prevention Maintenance Window
A “dummy” maintenance window for all agents set to some date in the future which will prevent machines from running the past deadline SUM Deployment Policy outside of their actual maintenance window. Caution: this maintenance window will prevent all machines from running any software distribution, unless override maintenance windows option is selected on the advertisement. *Important Safety Tip* set this prevention date far enough in the future that you will not likely ever reach it; or at least until you expect to leave your role and it’s someone else’s problem
We manage a large and dynamic population of servers and we offer choices of ~42 different maintenance windows (6 per day x 7 days). Servers can move maintenance windows at will and some machines receive patching extensions that have to be kept out of all targeted collections; maintaining these manually is not an option. I wrote some PowerShell tools that handle the translation of selected maintenance windows in a CMDB –> SCCM collections/advertisements; maybe a future blog topic.
Variables (Collection/Task Sequence) for Orchestration Control
Use collection variables and task sequence variables as program flow gateways for steps/groups inside the task sequence. Variables offer a simple way to assign programs to machines or even multiple programs using base variables. Variables can also collect execution return codes to postpone reboots or to send machines down a different set of steps/groups.
Looking for a good way to manage collection variables? Refer to my PowerShell library blog post about it.
Program Exit Code / Reboot Control
There aren’t many options available for controlling the task sequence reboot behavior at each step; any program that returns the universal 3010 code will cause the Task Sequence Engine to reboot the machine immediately after the step completes. I learned this lesson the hard way when we lined up five QFE/Hotfix installs and machines rebooted after each one.
It’s best to use a simple wrapper script to call the software installer and then manage the return code to mark a variable for reboot required and then pass back a 0 (success) return code. Also beware of native program return codes that map to irrelevant SCCM status messages – after a bit of pointless troubleshooting I learned that a QFE returned a WSUS exit code that equated to “Update Not Required” but the error message in ExecMgr.log said something like “File Not Found”.
Status Message Increase
Prepare for an increase in status messages during patching windows, especially as you add steps/groups into the task sequence. Agents will send status messages (sometimes multiple) for each step/group they process. While this yields a new level of detail for your reporting it also increases the load on the infrastructure/network during patching windows.
Hopefully I’ve peaked your interest in switching to the Task Sequence Engine for patch management; while it definitely adds an extra layer (or two!) of complication the benefits gained in orchestration, reporting and general extensibility (to name a few) make it worth the investment…