SCCM Backup Fails

We had an issue were 2 of our SCCM servers stopped having successful backups.

If I rebooted the server I could get 1 successful backup, but then the next backup would fail and would continue to fail until I rebooted, then the process would repeat.

This is the error I was getting in the smsbkup.log

Error: Failed to backup \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy7\SQL\SCCM_DB\SMS_XXX.mdf up to D:\SCCM_Backup\XXXBackup\SiteDBServer: CopyFile failed: Insufficient system resources exist to complete the requested service.      SMS_SITE_BACKUP          10/8/2009 9:08:22 AM      14456 (0x3878)

STATMSG: ID=5043 SEV=E LEV=M SOURCE="SMS Server" COMP="SMS_SITE_BACKUP" SYS=SCCMXXX SITE=XXX PID=3944 TID=14456 GMTDATE=Thu Oct 08 13:08:22.998 2009 ISTR0="\\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy7\SQL\SCCM_DB\SMS_XXX.mdf" ISTR1="D:\SCCM_Backup\XXXBackup\SiteDBServer" ISTR2="" ISTR3="" ISTR4="" ISTR5="" ISTR6="" ISTR7="" ISTR8="" ISTR9="" NUMATTRS=0          SMS_SITE_BACKUP  10/8/2009 9:08:22 AM      14456 (0x3878)

Failed to copy file(s) XXXBackup\SiteDBServer.  SMS_SITE_BACKUP  10/8/2009 9:08:22 AM   14456 (0x3878)

Error: Backup Failed  for Component - SMS_XXX.         SMS_SITE_BACKUP          10/8/2009 9:08:22 AM      14456 (0x3878)

In the SCCM Component status –> SMS_SITE_BACKUP you will see this error

Error    Milestone    XXX    10/14/2009 2:04:44 AM    OASCCM01    SMS_SITE_BACKUP    5043    SMS Site Backup was unable to back up file system object \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy11\SQL\SCCM_DB\SMS_XXX.mdf to D:\SCCM_Backup\XXXBackup\SiteDBServer.   

Possible cause: \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy11\SQL\SCCM_DB\SMS_XXX.mdf is not readable, or D:\SCCM_Backup\XXXBackup\SiteDBServer is not writable. 

Solution: Adjust file attributes and/or permissions of \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy11\SQL\SCCM_DB\SMS_XXX.mdf and/or D:\SCCM_Backup\XXXBackup\SiteDBServer.   

Possible cause: The destination directory D:\SCCM_Backup\XXXBackup\SiteDBServer did not already exist, and SMS Site Backup was unable to create it. 

Solution: Verify that a regular file with the same name as D:\SCCM_Backup\XXXBackup\SiteDBServer or one of its parent directories does not exist, or replace D:\SCCM_Backup\XXXBackup\SiteDBServer with an existing directory.   

Possible cause: \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy11\SQL\SCCM_DB\SMS_XXX.mdf or D:\SCCM_Backup\XXXBackup\SiteDBServer are on remote computers that SMS Site Backup does not have sufficient access rights to administer. 

Solution: Verify that the Site System Connection accounts are properly configured to allow SMS to administer the remote computer.   

Possible cause: An application is using D:\SCCM_Backup\XXXBackup\SiteDBServer or some component of its path (for example, Windows Explorer may be viewing D:\SCCM_Backup\XXXBackup\SiteDBServer's parent directory.)

Solution: Verify that D:\SCCM_Backup\XXXBackup\SiteDBServer and its parent directories are not in use by any application.   

Possible cause: Source \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy11\SQL\SCCM_DB\SMS_XXX.mdf being copied to D:\SCCM_Backup\XXXBackup\SiteDBServer may cause a recursive operation 

Solution: Verify that destination folder does not exist within the source folder.   

Possible cause: Source \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy11\SQL\SCCM_DB\SMS_XXX.mdf contains some file(s) that is opened exclusively by some application. 

Solution: Retry backup

*****************************************************************************************

NONE OF THESE SOLUTIONS WORK

When your backup log has this error try this solution.

Software setup

Server 2003 SP2 R2

SCCM 2007 SP1 R2

SQL 2005 SP3

Following solutions were applied

First apply this patch and reboot

http://support.microsoft.com/default.aspx?scid=kb;EN-US;940349

Then apply this reg key value at 60 and reboot

http://support.microsoft.com/default.aspx?scid=kb;EN-US;304101

Registry setting 1

1. Click Start, click Run, type regedit in the Open box, and then click OK.

2. Locate and then click the following registry subkey:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\Memory Management

3. On the Edit menu, point to New, and then click DWORD Value.

4. Type PoolUsageMaximum as the entry name, and then press ENTER.

5. Right-click PoolUsageMaximum, and then click Modify.

6. Click Decimal.

7. In the Value data box, type 60, and then click OK.
Important

• Use 60 as your initial value. If your backup does not succeed, use 40 as your value. If that does not work, you must change the behavior of your backup program to reduce the demand of paged pool. If the value works, you may want to increase the value by approximately 25 percent until the backup does not work. If the backup is unsuccessful, use the second registry setting that is described in this article.
• Make sure that the value for this registry setting is not more than 60.
• If you are using the /3GB switch, use 40 as your initial setting. Note that this value is a percentage value.

8. Quit Registry Editor.

9. Restart your computer.

Because you must test these settings during the most stressful backups, you may have to wait a month for a whole backup cycle to complete if you are not sure which backup consumes the most resources. Because of this situation, Microsoft recommends that you test low values first. For more information, click the following article number to view the article in the Microsoft Knowledge Base:

312362 (http://support.microsoft.com/kb/312362/ ) Server is unable to allocate memory from the system paged pool

Hope this helps someone else out.