When you are in the process of implementing Windows Vista in an enterprise environment, you may run into the following situation:
You have a logon script that writes a file to disk or queries AD with ADSI. For no apparent reason the script hangs for minutes before it finishes without any error message. On further investigation you will find no issues regarding the network, memory or disk resources. When you manually start the script it always runs fast without issues. What is happening here?
Get ready for a long story.
Windows Vista has a number features that are supposed to improve the user experience during system startup and logon. One of these is Boxing. Boxing is a feature that makes sure that programs that start while you logon are no longer able to start fighting for all available resources eating all disk resources and CPU. Usually programs like the Adobe Updater, Winzip Quicklauncher, iTunes quicklaunch or whatever these programs are called, are not really of interest for the user in terms of how fast they startup. But they will slow down the system significantly when Windows threats them just as equal as you manually try to start after logon (like Outlook or Word for example). Windows Vista lowers the thread priority level for startup applications to " Below Normal" and will not allow them to raise their thread priority during the first 60 seconds after logon. In this way Vista slows down the execution of these applications and leaves resources available for programs that the user actually wants to start.
So far so good. Vista uses this mechanism for programs started from the following locations:
- Startup folder in the Start Menu
- Windows\CurrentVersion\Run key in the Registry
- Scheduled tasks initiated at Logon
- Defined in Group Policy in: User Configuration | Administrative Templates | System| Logon | Run these programs a user logon
Windows Vista actually lowers the thread priority level of these applications to BelowNormal and disables their normal ability to bump up their own thread level during the first minute after logon. Maarten from the Vista product team wrote a little piece about this behavior in their blog. At the same time Vista lowers the application' s I/O priority level to "very low I/O priority level". This is a new feature that did not exist before Windows Vista. Normally a process that writes a file to disk does not directly access the disk, but uses a mechanism called the file cache. When using the file cache, a program writes data into a RAM based memory cache first. Then Windows will take care of actually writing the data to disk when it's convenient for the system. Writing data to the file cache is really fast and programs usually hardly wait for data being written to the file cache. Processes running at the very low I/O priority level do not make use of the file cache, but write their data directly to disk when the Windows file system driver thinks it is convenient. This makes writing data to disk very very slow while the application is being "boxed".
How are logon scripts affected?
Logon scripts defined in Group Policy by default run asynchronous and invisible to the end user. Asynchronous logon scripts run in parallel with the user logging on. In this way other processes do not have to wait until the script(s) finishes. The default behavior of logon scripts imply that the scripts are actually background processes and will be handled as such in terms of thread and I/O priority. This means that logon script by default will be started at the "Low" priority thread level and the " Very low I/O priority" level. This is also true when you decide to run the script visible.
In the case of our logon script, not being able to use the file cache severely harms the performance of the script when it writes data to the local disk.
Where the thread priority of the first category of startup programs changes to normal after 60 seconds, the thread priority of logon scripts defined Group Policy stays "Low" forever just as the I/O priority level that stays "very low".
Scripts running at "very low i/o priority" has serious impact on disk write operations initiated from the script. Writing data does not perform as fast as the process expects. This causes the script to try a write operation and then see that it does not complete within the expected time, then wait for a certain time (50ms) and try again. This is very clear when you monitor the behavior of such a script with Process Monitor (from Sysinternals on the MS website). Process Monitor reports the script is running with very low IO priority and is not allowed to use fast file IO (using the file cache). Because of the very slow file IO you also see a lot of File Lock Conflict messages caused by subsequent write operations that fail because the earlier write operation did not end within the expected time frame.
Why is ADSI harmed by the very low IO priority thread level
It is less obvious that a script is writing to disk when you are using ADSI to collect data from AD. When the script requests information from AD, a local copy of the AD schema may be locally cached in the user profile as %userprofile%\\AppData\Local\Microsoft\Windows\SchCache\<forest_name>.sch, where <forest_name> is replaced with the fqdn of the AD forest. Windows Vista should only do this the first time a user logs on to the domain and used ADSI to query information from the domain, but I have seen cases where the Schema cache was rebuild every time at logon. Writing the schema cache (~ 1 MB) to disk can take 6 to 15 minutes.
Is there a workaround?
It is possible to disable boxing for programs initiated from the Startup folder or the Run key in the Registry. To accomplish this, the following registry key must be changed:
Value: Delay_Sec = 0 (default value = 60)
Changing the value is a bit hard because the key is owned by TrustedInstaller and Administrators only have Read access to the key. To overcome this hurdle you must first take ownership of the key.
Unfortunately programs initiated from logon scripts defined in Group Policy are not affected by this key.
If you need to have your logon scripts run at a different I/O level, you have the following options:
Both options have a their own issues:
In the end, what appeared to be a bug, seems to be a feature. In my opinion, an asynchronous logon script that runs visible is no longer a background process and should not be running a low or very low I/O and thread levels. I also have my doubts about the benefits of very low I/O priority and the way it works out for lots of applications or scripts that slow down to crawl speed without warning.