We've a 64bit Windows Server 2008 running Remote Desktop.
The application lsm.exe (the local session manager) appears to be leaking memory. Although the memory usage is quite low when the server is rebooted, this continues to climb until people can no longer log in.
The server has no audio card and does not have any AV software installed. The server is fully service packed. (Service pack 2)
It has 8GB memory. 60 users regularly use it. There seem to be no relevant event log messages. The login does not start explorer. It starts a "login.vbs" script. This login script creates several wsShell and wScript.Network objects, but even if I comment these out, I still get the same behaviour. (note - I swapped the .vbs script for an executable. No difference)
There are multiple applications installed, but the only one of note is : perfectbackup - A java based backup solution.
The memory leak seems to happen when people log in / log out of Remote Desktop. e.g. Logging in will change the lsm memory size from 21,180K to 21,512K. Logging out will change the usage from 21,512K to 12,668K. This may go down later on, but the general movement is upwards. Also - This will go up faster, and stay up if I log in / out several times.
What could be causing this, and how would I fix it?
How many sessions are generally active on your machine? If users are disconnecting their sessions, instead of logging off, that can cause the kind of problems you are seeing. One thing I've seen done to resolve this is to force disconnected sessions to log out after a short period of inactivity (1 to 2 hours) - that has resolved this type of issue for me in the past.
Perhaps because you are using a non windows shell you aren't actually properly logging out of terminal server. Forcing disconnections might not be enough?
Try adding to the end of the script:
Just a wild guess (and yes 4 is forced logoff ( http://msdn.microsoft.com/en-us/library/aa394058%28VS.85%29.aspx )
This looks like it could be related to the problem you're experiencing:
http://www.windows-now.com/blogs/kmkenney/archive/2007/05/21/workaround-for-lsm-exe-handle-and-memory-leak.aspx
Although the server has no audio card, maybe this could involve RDP's remote audio playback capability.
You should try disabling remote audio when connecting to the terminal server and see if the memory leak disappears.
Also, an additional troubleshooting step you should take: does this happen when logging on locally to the server's console, or only when using Terminal Server?
Found the following on the microsoft forums.
"The XP/2003 terminal session manager needs to be closed once every few hours. This resets the lsm.exe memory footprint. We monitor lsm.exe memory usage with SCOM and we see an immediate drop in memory usage when we close ts managers on XP or 2003. The 2008 ts manager doesn't seem to cause this problem."
http://social.technet.microsoft.com/Forums/en-US/winserverTS/thread/78b75882-870f-4a4a-8c4d-f8690c7c124a