How to troubleshoot Terminal Services connection problems

Recently, I was asked to resolve a problem where a large terminal server farm was having some unusual issues. The symptom was that as users connected to various terminal servers they would hit a connection limit. The problem hit all of the servers in a random fashion, but it seemed that they would allow only 45 connections.

The 46th connection was refused, and because the terminal servers had been in use for some time, it was difficult to explain why this behavior started.

In diagnosing the problem, there were a number of things to consider. Of course, it could be a resource limit on the servers – , I/O, perhaps even a network resource because they were all at a single site. I determined that:

• Event 1006 was logged in the system log.
• There were no Citrix components involved.
• The connection failure was not associated with a server hang.
• A reboot of the server did not resolve the problem.
• All servers, at one time or another, were affected.
• The beginning of the problem could not be tied to an application of any hotfixes or other patches.
• Because it was quite definite that the 46th connection was refused, there had to be a limit set somewhere.
• Group Policy under computer configuration\administrative templates\windows components \ terminal services\limit maximum number of connections was not defined.

Microsoft KB 923630 provides components for Gdi32.dll and Win32k.sys that resolves problems where a Terminal Server refuses connections, but this hotfix was quite old. Because these systems were at SP2, the components were newer than the hotfix.

However, KB 908912 states that the maximum connections to a terminal server is controlled by a registry key. The registry value is located at HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\TerminalServer, and is a Dword Value, MaxOutstandingConnect. Note that this value must be created. The KB, however, is not clear on what the value should be set to. It states:

Note: The registry value should be set to the appropriate threshold to achieve the required connection detection. For example, if 50 connections are required, set the value to 32 in hexadecimal format.

We needed many more connections than 50, and the 64-bit servers have the horsepower to do it. Because our limit of 45 connections seemed in the ballpark of the 50 in the example, it seemed reasonable to pursue this. We found that the default seemed to be 50. We raised the threshold to 100, and the connections stopped at 95. Set it to 75, and we could only get 70 connections.

Apparently Microsoft designed this feature to help regulate Terminal Services connections and prevent them from overtaxing server resources. Too bad the company didn't do a better job explaining it. I have no idea why we could get only five fewer connections than the value we set, but it was reproducible.

Because the connections do take up resources, we monitored the terminal servers as we modified this value. But our servers easily handled the load and we ended up with a maximum connection setting that the server could handle and it provided the number of connections that we wanted. This had to be set on each terminal server, but it did solve our problem.

The point here is not to say that this registry key is one-stop shopping for solving terminal server connection limit problems, but it can be a factor. When troubleshooting this issue, look at the bullet points in this article for other solutions.

It is important to identify the scope of the problem -- all servers or just a few -- if the limit seems to be consistent and if resources are depleted. Just setting this value without good troubleshooting of the problem will not necessarily solve it.

Gary Olsen is a systems software engineer for Hewlett-Packard in Global Solutions Engineering. He authored  Windows 2000: Active Directory Design and Deployment and co-authored Windows Server 2003 on HP ProLiant Servers. Gary is a Microsoft MVP for Windows Server-File Systems.