Questions[response-time](server)

I have a set of very beefy, very underutilized servers which are running a couple virtual machines with Windows Server 2008 R2 and IIS 7.5.

The problem: Sometimes requests take a very long time to be processed. The user sees their browser spinning, apparently not getting any response from IIS.

Some stats & attempts at resolving:

• The load on the servers (host and VM) is negligible. CPU never goes over 5%, there are 10+ GB of RAM available, everything is connected via MPIO to a fast SAN.
• I get between 30 and 50 requests per second, a mix of dynamic and static content, only GET and POST. Most hits are cached (80% hit rate), so the load on the fabric/SAN/IO is close to nothing.
• TCP offloading is disabled on the host and virtual machine, both on the network adapter and via disabling TCP chimney
• The web applications are running in ASP.NET 4.0 Integrated Mode. They do not make long-running calls to third party web services or the like.
• I have tried both the processModel autoConfig, as well as setting maxWorkerThreads, maxConnections, maxIOThreads etc. to very high numbers with no difference.
• Database querys are all completing in less than 1 second. I profiled for an entire day and didn't capture a single query that would take longer.
• I've looked at tons of performance monitor counters; the ASP.NET queues and Application queues are always empty, nothing appears to get queued (which wouldn't make sense anyways considering that the server isn't breaking a sweat at all)
• I've identified using appcmd list requests that sometimes requests get 'stuck' for 20-60 seconds in IIS' "SendResponse" stage. This is the only thing I could find so far that would make any sense as to why we're seeing users getting stuck when navigating around. Note that most requests are getting processed fast though, it looks just like random requests from different application pools get stuck here.

Any idea what else I can look at? What would cause requests getting stuck in the 'SendResponse' stage in IIS?

I have an Ubuntu server machine running Apache and MYSQL. System and version info is as follows: Linux kernel 3.0.0.-12 Apache/2.2.20 MySQL Ver 14.14.Distrib 5.1.58

I am running a few websites on this server, some HTML only, some PHP/MySQL. THe [problem is that response time is very slow, both on static as well as the dynamic sites. Sometimes it takes more than 10 seconds before a response is given, this makes the sites very slow and almost unusable.
The problem occurs even when requesting from the local network. I have added the involved subdomains to my /etc/hosts file, and abolve all the problem is not solved by using IP numbers instead of URL's. So there is no DNS lookup issue.
I have modified the log format by showing the response times and sometimes a files takes 12 seconds to be served, see the jquery~.js file in the example screenshot.
I have no explanation for this extremely long response time, but is is not even the only issue here, some other files takes a long time to be served too, but do not show a long response time in the log file.
So probably different tissues are involved here. I cannot find a solution until now, any suggestions??? THanx in advance, Klaas

link to screenshot picture from access logfile

Some extra configuration info:

apache2.conf (comment is removed)

LockFile ${APACHE_LOCK_DIR}/accept.lock
PidFile ${APACHE_PID_FILE}
Timeout 300
KeepAlive On
MaxKeepAliveRequests 100
KeepAliveTimeout 5
<IfModule mpm_prefork_module>
    StartServers          5
    MinSpareServers       5
    MaxSpareServers      10
    MaxClients          150
    MaxRequestsPerChild   0
</IfModule>

<IfModule mpm_worker_module>
    StartServers          2
    MinSpareThreads      25
    MaxSpareThreads      75 
    ThreadLimit          64
    ThreadsPerChild      25
    MaxClients          150
    MaxRequestsPerChild   0
</IfModule>

<IfModule mpm_event_module>
    StartServers          2
    MinSpareThreads      25
    MaxSpareThreads      75 
    ThreadLimit          64
    ThreadsPerChild      25
    MaxClients          150
    MaxRequestsPerChild   0
</IfModule>

User ${APACHE_RUN_USER}
Group ${APACHE_RUN_GROUP}

AccessFileName .htaccess

<Files ~ "^\.ht">
    Order allow,deny
    Deny from all
    Satisfy all
</Files>

DefaultType text/plain
HostnameLookups Off

ErrorLog ${APACHE_LOG_DIR}/error.log
LogLevel warn

Include mods-enabled/*.load
Include mods-enabled/*.conf
Include httpd.conf
Include ports.conf
LogFormat "%v:%p %h %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\"" vhost_combined
LogFormat "%h %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\" %T/%D" combined
LogFormat "%h %l %u %t \"%r\" %>s %O" common
LogFormat "%{Referer}i -> %U" referer
LogFormat "%{User-agent}i" agent
Include conf.d/
Include sites-enabled/

And the virtual hostfile for one of the slow sites, in fact it is pretty straightforward...

<VirtualHost *:80>
    ServerAdmin [email protected]
        ServerSignature EMail
        ServerName toenjoy.drsklaus.nl
    DocumentRoot /var/www/toenjoy.drsklaus.nl
    <Directory />
        Options FollowSymLinks
        AllowOverride None
    </Directory>
    <Directory /var/www/toenjoy.drsklaus.nl/>
        Options Indexes FollowSymLinks MultiViews
        AllowOverride AuthConfig
                AuthType Basic
                AuthName "To Enjoy"
                AuthUserFile /etc/.htpasswd
                Require user petraaa
        Order allow,deny
        allow from all
    </Directory>

    ScriptAlias /cgi-bin/ /usr/lib/cgi-bin/
    <Directory "/usr/lib/cgi-bin">
        AllowOverride None
        Options +ExecCGI -MultiViews +SymLinksIfOwnerMatch
        Order allow,deny
        Allow from all
    </Directory>

    ErrorLog /var/log/apache2/error.log

    # Possible values include: debug, info, notice, warn, error, crit,
    # alert, emerg.
    LogLevel warn
    CustomLog /var/log/apache2/access.log combined

    Alias /doc/ "/usr/share/doc/"
    <Directory "/usr/share/doc/">
        Options Indexes MultiViews FollowSymLinks
        AllowOverride None
        Order deny,allow
        Deny from all
        Allow from 127.0.0.0/255.0.0.0 ::1/128
    </Directory>
</VirtualHost>

And the output of free -m:

klaas@ubuntu-server:/etc/apache2$ free -m
             total       used       free     shared    buffers     cached
Mem:          1997       1401        595          0        144       1017
-/+ buffers/cache:        238       1758
Swap:         2035          0       2035

and I have no indication that swapping occurs on the moments the site is slow. I have runned top and it does not appear to be a CPU issue. I have the impression that the spawning of a apache thread could maybe be the bottleneck but it is just a suggestion. Maybe this gives some extra information!

EDIT: The problem seemed to be gone for some time but occurs again! And not only with Apache, also connecting using SSH takes a tremendous time, sometimes it takes up to 15 seconds before the keyphrase is asked for. Also scp works very slowly. The behavious is really unpredoctable and makes the server very hard to use. Any ideas...?

I've recently moved a client's website (using the concrete5 CMS) over to a VPS running Gentoo, Apache 2.2, PHP5 and MySQL 5 and I've noticed that Apache response times are pretty bad (it was the same on the old server), sometimes way up to 8-9 seconds, but more often between 300ms and 3 seconds (towards 300ms I don't mind). I know it's not network latency, since the server has a ping (from my location) of around 30ms.

Here's an example of the times (you can see it's snappy after the initial wait):

I'm running APC (although I'm not sure that's working right...) and SuExec. Apache modules are:

 core_module (static)
 authn_file_module (static)
 authn_default_module (static)
 authz_host_module (static)
 authz_groupfile_module (static)
 authz_user_module (static)
 authz_default_module (static)
 auth_basic_module (static)
 include_module (static)
 filter_module (static)
 deflate_module (static)
 log_config_module (static)
 env_module (static)
 expires_module (static)
 headers_module (static)
 setenvif_module (static)
 version_module (static)
 ssl_module (static)
 mpm_prefork_module (static)
 http_module (static)
 mime_module (static)
 status_module (static)
 autoindex_module (static)
 asis_module (static)
 info_module (static)
 suexec_module (static)
 cgi_module (static)
 negotiation_module (static)
 dir_module (static)
 actions_module (static)
 userdir_module (static)
 alias_module (static)
 rewrite_module (static)
 so_module (static)
 suphp_module (shared)

and PHP modules are:

bcmath
calendar
ctype
curl
db
dbase
domxml
exif
ftp
gd
gettext
iconv
imap
mbstring
mcrypt
mime_magic
mysql
openssl
overload
pcre
posix
session
standard
sysvsem
sysvshm
tokenizer
xml
xslt
zlib

I've got gzip enabled on all relevant files.

Apache is running using prefork, and the settings in httpd.conf are:

<IfModule prefork.c>
StartServers         10
MinSpareServers      10
MaxSpareServers      20
MaxClients           250
MaxRequestsPerChild  4000
</IfModule>

HostnameLookups Off

I've noticed that pages which (I think) are database-heavy, such as the CMS's Dashboard, are usually slower. I thought this might mean MySQL could be optimised. I wondered also about Apache modules - I get confused between mod_php5, mod_cgi, mod_fastcgi etc etc - there is conflicting advice all over the net as to the best one to use.

Here's the output of MySQLTuner:

-------- General Statistics --------------------------------------------------
[--] Skipped version check for MySQLTuner script
[OK] Currently running supported MySQL version 5.0.44-log
[OK] Operating on 64-bit architecture

-------- Storage Engine Statistics -------------------------------------------
[--] Status: -Archive -BDB -Federated -InnoDB -ISAM -NDBCluster
[--] Data in MyISAM tables: 35M (Tables: 161)
[!!] Total fragmented tables: 15

-------- Security Recommendations  -------------------------------------------
[OK] All database users have passwords assigned

-------- Performance Metrics -------------------------------------------------
[--] Up for: 3d 21h 44m 16s (293K q [0.868 qps], 1K conn, TX: 135M, RX: 90M)
[--] Reads / Writes: 99% / 1%
[--] Total buffers: 58.0M global + 1.6M per thread (100 max threads)
[!!] Maximum possible memory usage: 219.7M (93% of installed RAM)
[OK] Slow queries: 0% (0/293K)
[OK] Highest usage of available connections: 2% (2/100)
[OK] Key buffer size / total MyISAM indexes: 16.0M/20.9M
[OK] Key buffer hit rate: 99.6% (5M cached / 21K reads)
[!!] Query cache is disabled
[OK] Sorts requiring temporary tables: 0% (0 temp sorts / 3K sorts)
[!!] Temporary tables created on disk: 47% (2K on disk / 5K total)
[!!] Thread cache is disabled
[!!] Table cache hit rate: 6% (64 open / 1K opened)
[OK] Open file limit used: 12% (128/1K)
[OK] Table locks acquired immediately: 100% (356K immediate / 356K locks)

-------- Recommendations -----------------------------------------------------
General recommendations:
    Run OPTIMIZE TABLE to defragment tables for better performance
    Reduce your overall MySQL memory footprint for system stability
    Enable the slow query log to troubleshoot bad queries
    When making adjustments, make tmp_table_size/max_heap_table_size equal
    Reduce your SELECT DISTINCT queries without LIMIT clauses
    Set thread_cache_size to 4 as a starting value
    Increase table_cache gradually to avoid file descriptor limits
Variables to adjust:
  *** MySQL's maximum memory usage is dangerously high ***
  *** Add RAM before increasing MySQL buffer variables ***
    query_cache_size (>= 8M)
    tmp_table_size (> 32M)
    max_heap_table_size (> 16M)
    thread_cache_size (start at 4)
    table_cache (> 64)

I noticed when a DB-heavy page was loaded, CPU usage spiked at 57% (using top) - to me that suggests there's either some badly-optimised MySQL stuff or caching is absolutely necessary to speed this setup up.

Any help would be much appreciated!

I have a nginx reverse proxy. The server is close to serving 600-700 requests per second. I have a Munin HTTP load time plugin which is outputting this:

http://monitor.wingify.com/munin/visualwebsiteoptimizer.com/lb1.visualwebsiteoptimizer.com-http_loadtime.html

Now, the problem is I am seeing some spikes in the graph. Expected response times should always be under 200ms. I am keeping an eye on syslog and messages but I am unable to figure out the actual cause of this. I was wondering if there is any good HTTP response time profiling system which I can install / embed with this nginx server and get a detailed reports / logs on the breakup of time taken by different things and what exactly is the cause of the spikes.

The profiling system would also help me understand bottlenecks and how can I further optimize the latency.

Most important right now is to investigate the cause of the spikes in the HTTP load time graphs (similar pattern is reported by external monitors - Pingdom) and to fix it to get consistent response times

Thanks

I'm wondering what a maximum (practical) ping response time might be. As far as I know, there isn't a max defined anywhere (TTL, but that's hops, not time). As I think about it, I'm not sure I've ever seen a ping response time of more than a second or so. But as far as I know, there is nothing to stop a remote host from waiting (or being really busy) and not sending the response back for a few seconds.

As a simple data point, I just pinged a number of servers around the world and the worst time I could find was 350ms.