I have a 3 servers setup, one frontend HAproxy sending requests to 2 backend servers.
The backend servers use identical hardware and also identical software (Debian 6.0.6). They are similarly configured also receive roughly the same number of requests to process (HAproxy takes care of this).
Despite of this one has constantly a higher load than the other and it seems to be related to hard disks as far as I can tell by using iotop
The processes that seem to read most are php-fpm processes.
Any advices on where I could look would be hopefully helpful.
After much digging I figured out the main reason why one server was working much better than the other:
one had 1024 for
/sys/block/sda/queue/nr_requests
while the other one had the default value which was 128.