Shaun Cassells at MyITForum.com

SMS 2003 and ConfigMgr 2007, PowerShell, Scripting, Finance, Fitness and Fun

News

Locations of visitors to this page

Boxing and the case of the slow or hanging logon script in Vista

http://bink.nu/news/boxing-and-the-case-of-the-slow-or-hanging-logon-script-in-vista.aspx
 

When you are in the process of implementing Windows Vista in an enterprise environment, you may run into the following situation:

You have a logon script that writes a file to disk or queries AD with ADSI. For no apparent reason the script hangs for minutes before it finishes without any error message. On further investigation you will find no issues regarding the network, memory or disk resources. When you manually start the script it always runs fast without issues. What is happening here?

Get ready for a long story.

Windows Vista has a number features that are supposed to improve the user experience during system startup and logon. One of these is Boxing. Boxing is a feature that makes sure that programs that start while you logon are no longer able to start fighting for all available resources eating all disk resources and CPU. Usually programs like the Adobe Updater, Winzip Quicklauncher, iTunes quicklaunch or whatever these programs are called, are not really of interest for the user in terms of how fast they startup. But they will slow down the system significantly when Windows threats them just as equal as you manually try to start after logon (like Outlook or Word for example). Windows Vista lowers the thread priority level for startup applications to " Below Normal" and will not allow them to raise their thread priority during the first 60 seconds after logon. In this way Vista slows down the execution of these applications and leaves resources available for programs that the user actually wants to start.

So far so good. Vista uses this mechanism for programs started from the following locations:

  • Startup folder in the Start Menu
  • Windows\CurrentVersion\Run key in the Registry
  • Scheduled tasks initiated at Logon
  • Defined in Group Policy in: User Configuration | Administrative Templates | System| Logon | Run these programs a user logon

Windows Vista actually lowers the thread priority level of these applications to BelowNormal and disables their normal ability to bump up their own thread level during the first minute after logon. Maarten from the Vista product team wrote a little piece about this behavior in their blog. At the same time Vista lowers the application' s I/O priority level to "very low I/O priority level". This is a new feature that did not exist before Windows Vista. Normally a process that writes a file to disk does not directly access the disk, but uses a mechanism called the file cache. When using the file cache, a program writes data into a RAM based memory cache first. Then Windows will take care of actually writing the data to disk when it's convenient for the system. Writing data to the file cache is really fast and programs usually hardly wait for data being written to the file cache. Processes running at the very low I/O priority level do not make use of the file cache, but write their data directly to disk when the Windows file system driver thinks it is convenient. This makes writing data to disk very very slow while the application is being "boxed".

How are logon scripts affected?

Logon scripts defined in Group Policy by default run asynchronous and invisible to the end user. Asynchronous logon scripts run in parallel with the user logging on. In this way other processes do not have to wait until the script(s) finishes. The default behavior of logon scripts imply that the scripts are actually background processes and will be handled as such in terms of thread and I/O priority. This means that logon script by default will be started at the "Low" priority thread level and the " Very low I/O priority" level. This is also true when you decide to run the script visible.

In the case of our logon script, not being able to use the file cache severely harms the performance of the script when it writes data to the local disk.

Where the thread priority of the first category of startup programs changes to normal after 60 seconds, the thread priority of logon scripts defined Group Policy stays "Low" forever just as the I/O priority level that stays "very low".

Scripts running at "very low i/o priority" has serious impact on disk write operations initiated from the script. Writing data does not perform as fast as the process expects. This causes the script to try a write operation and then see that it does not complete within the expected time, then wait for a certain time (50ms) and try again. This is very clear when you monitor the behavior of such a script with Process Monitor (from Sysinternals on the MS website). Process Monitor reports the script is running with very low IO priority and is not allowed to use fast file IO (using the file cache). Because of the very slow file IO you also see a lot of File Lock Conflict messages caused by subsequent write operations that fail because the earlier write operation did not end within the expected time frame.

Why is ADSI harmed by the very low IO priority thread level

It is less obvious that a script is writing to disk when you are using ADSI to collect data from AD. When the script requests information from AD, a local copy of the AD schema may be locally cached in the user profile as %userprofile%\\AppData\Local\Microsoft\Windows\SchCache\<forest_name>.sch, where <forest_name> is replaced with the fqdn of the AD forest. Windows Vista should only do this the first time a user logs on to the domain and used ADSI to query information from the domain, but I have seen cases where the Schema cache was rebuild every time at logon. Writing the schema cache (~ 1 MB) to disk can take 6 to 15 minutes.

Is there a workaround?

It is possible to disable boxing for programs initiated from the Startup folder or the Run key in the Registry. To accomplish this, the following registry key must be changed:

Key: HKLM\Software\Microsoft\Windows\CurrentVersion\Explorer\Advanced\DelayedApps

Value: Delay_Sec = 0 (default value = 60)

Changing the value is a bit hard because the key is owned by TrustedInstaller and Administrators only have Read access to the key. To overcome this hurdle you must first take ownership of the key.

Unfortunately programs initiated from logon scripts defined in Group Policy are not affected by this key.

If you need to have your logon scripts run at a different I/O level, you have the following options:

  • Run the scripts synchronous instead of asynchronous
  • Use another mechanism to start the script:
    • Startup folder in the Start Menu
    • Windows\CurrentVersion\Run key in the Registry
    • Scheduled tasks initiated at Logon
    • Use the policy: User Configuration | Administrative Templates | System| Logon | Run these programs a user logon

Both options have a their own issues:

  • Issues when running logon scripts synchronous:
    • Running logon scripts synchronous will kill all child processes started from the script when the script ends.
    • The user will not get to see the desktop until the script ends. If something goes wrong and the script does not end, the end user will never see the desktop.
  • Issues when using one of the other startup options:
    • Harder to delegate, because with logon scripts each GPO has its own folder structure for scripts.
    • These scripts are still being boxed for the first 60 seconds. Any script that writes to disk will be slowed down during the first minute unless you completely disable boxing (which is not recommended by Microsoft).

Conclusion

In the end, what appeared to be a bug, seems to be a feature. In my opinion, an asynchronous logon script that runs visible is no longer a background process and should not be running a low or very low I/O and thread levels. I also have my doubts about the benefits of very low I/O priority and the way it works out for lots of applications or scripts that slow down to crawl speed without warning.

Posted: Feb 05 2008, 02:17 PM by scassells | with 3 comment(s)
Filed under: , ,

Comments

Windows Vista News said:

Interesting: myitforum.com

# February 5, 2008 5:30 PM

Andrei Ungureanu's Blog said:

In fiecare zi mai aflu ceva nou despre Vista (la fel si despre XP si W2K3). Azi am aflat de termenul

# May 7, 2008 11:52 AM

K.E.Jones said:

Hi,

Thanks, this was a really good read (enough that I joined to say this!).

Not only did I learn some things I didn't know (such as using the scheduler to co-ordinate things) but your conclusions also caught my eye.

I really totally agree with your comments on visible processes. What exactly is the point of throttling something that inherently begs for a real-time presence?

IMNSHO,  It's not unreasonable in a work environment to want to display some form of message from a logon/startup script. Vista's behaviour at preserving the user experience works against itself when scripts attempt this.

Alongside the asynchronous and boxing changes MS have implemented, they've also changed how 'visible' the login scripts are.

The policy setting for "run scripts visibly" now also covers child processes and script output. It's a real shame because you now can't launch a script that provides its own flashy graphics without seeing that ugly DOS launch box :-(.

That's really a bit of downer on the smoothness of Vista's glossy startup process. I'd bet the Kixx community would like to kixx MS's butt right now ;-)

It's not nice to criticise negatively without at least trying to offer a positive solution so; if I was MS;

a) I'd add a third setting to logon scripts policies, "Run silent but allow visible children."

b) Shim everything into the scheduler with a good straightforward API for programmers. Oops, sounds like CRON to me...Maybe these Linux guys have a point ;-)

c) Allow more 'latitude' for signed scripts? Hopefully with support of Domain Admin authorised authenticode signatures people could avoid the hefty costs of having global signatures.

What do you think?

Have fun, and I hope I wasn't being too off-topic for this to be interesting to you!

Regards,

Keith

# June 26, 2008 4:46 PM