Find out the bottleneck for windows remote desktop server (Terminal server)

As Chris S mentioned, there are several things that can contribute to poor remote desktop performance. From my experience, these are the main causes, in order of likelihood.

Bandwidth
The #1 cause of poor performance with remote desktop is lack of bandwidth. Depending on exactly what is being done, a session can use anywhere from a a few Kbps to a few Mbps of bandwidth. My own tests have shown that scrolling through a PDF will use up to 3 Mbps. As available bandwidth decreases, so does perceived performance.

You first need to determine the bandwidth needs of your application. This requires testing in a controlled LAN environment, then measuring the bandwidth use as you perform normal tasks. I personally had success with NetLimiter on my personal workstation. You can also approach the problem from another angle, and use NetLimiter to force your connection speed down to whatever your WAN connection is rated at. This should give a good indication of what your remote users are seeing.

Once you know how much bandwidth your application wants, you need to determine whether it is the limiting factor. First, measure the available bandwidth between the client and the server. An excellent tool for this is iperf. I'll assume that you have sufficient bandwidth available during a controlled test.

Next, you will want to set up some kind of bandwidth monitoring to see whether user-reported problems correlate with spikes in traffic or other undesirables. My preference is to dump traffic from a switch or router into ntop, as it provides useful realtime and historic reports on bandwidth usage.

If you're encountering bandwidth problems, one easy change is to change the "Experience" settings on the remote desktop connection. Disable visual styles and animations, and many desktop operations will seem magically faster.

Latency
Another common issue with remote desktop connections is latency. There needs to be a reasonably quick round-trip time between the client and server, or people will be able to perceive a delay. As a rule of thumb, most people start to notice problems between 50 and 100 ms ping times.

Luckily, this is usually easy to diagnose. You can set up monitoring tools like SmokePing or PRTG Network Monitor to provide reports on the latency between your monitoring server and any other arbitrary host. You can even just use the built-in ping -t command for short sessions. Normally you want to locate the monitoring server on the same LAN as your remote desktop server, then set up monitoring against both the server and your clients. Try to correlate problem reports with incidents of high ping time.

If you're having problems with high ping times, use traceroute to find out where the delay is being introduced. If you determine that the problem resides within your own network, consider introducing QoS filtering to prioritize realtime traffic like Remote Desktop.

Also, be wary of anybody who is connecting over a wireless medium, whether that is 802.11 (WiFi), or worse, a satellite connection. Wireless connections are prone to environmental interference that can cause extreme latency problems under various conditions, and for varying periods of time. And using remote desktop through a satellite always sucks.

Local CPU or memory And finally, it's possible that your server is simply overburdened. Monitor the CPU and memory usage, especially during peak hours, to ensure that the server is capable of meeting requests in a timely fashion.

One of the tools mentioned above (PRTG) can be set up to monitor CPU and memory usage of a server over time, and can produce graphs that make it easy to correlate problem problem reports with specific faults.

Bonus tip: If your users are having trouble typing, especially with regard to modifier keys not applying properly, try changing your keyboard settings on the Remote Desktop connection shortcut so that Apply Windows key combinations is set to On the local computer.

Check out the RDS Best Practises Analyzer.