Jeff Gilbert's Web blog at myITforum.com

In Configuration Manager 2007, it is supported to install the site database on a SQL Server named instance and not just the default instance as it was in SMS 2003. Regardless of whether or not you use the default or a named instance of SQL Server to host the site database, a Service Principal Name (SPN) must be registered for the SQL Server service account in Active Directory to enable Kerberos authentication.

When the SQL Server service account is configured to use the local system account, the server will automatically publish the SPN for you. However, a SQL Server best practice is to change the startup account from local system to a domain user account to better secure the SQL Server instance. If you're using a domain user account to run the SQL Server service, you have to manually create the SPN for the account in Active Directory. Once created, you can view the SPNs registered using an ADSIEdit console.

Note: To use the SetSPN utility, or to open an ADSIEdit MMC console, you must first install the Microsoft Windows Server support tools. These tools are included in the support tools folder on both Windows 2000 Server and Windows Server 2003 CDs. To install the Windows Server support tools, navigate to \SUPPORT\TOOLS\ on the server's installation CD and run suptools.msi.

To register the SPN for the domain user account in Active Directory for the default instance of SQL Server (assuming you haven't changed the port it is listening on) you can use the following syntax:

Setspn -A MSSQLSvc/<SQL Server name>:1433 <domain>\<user>

The trick here is that you have to do this twice. You need to register the SPN for both the SQL Server computer's NetBIOS name and FQDN to allow Setup to succeed and for the site to operate properly after it is installed.

When using a SQL Server named instance to host the site database, and using a domain user account as its start up account, you must register the SPN for the named instance in Active Directory. When registering the SPN for a SQL Server named instance, the syntax is the same as creating one for the default instance--the named instance is detected by the port number it responds to and is not specified as part of the SQL Server name (just like IIS and the default Web site, you can't use the same port that the default instance uses for a named instance of SQL Server).

So, the command to register the SPN for the domain user account running a named instance using port 1400 (1400 is just a random port I picked out of my head, not a recommendation) would be:

Setspn -A MSSQLSvc/<SQL Server name>:1400 <domain>\<user>

Once again, you have to do this twice, once for the SQL Server's NetBIOS name and once for the SQL Server's FQDN.

Troubleshooting Tips
There is a known issue when running the stand-alone prerequisite checker on named instances in case you see an error when running that that says you must specify a valid NetBIOS host name. To test the named instance before beginning the install, you can use the Setup command line:

Setup /Prereq /Pri /SQL <SQL Server Name\Instance Name>

Also, when running Setup to install the site database on a remote SQL Server named instance, the SQL Server browser service must be running when Setup goes looking to validate it or else Setup will fail.