What is the best sysctl.conf configuration for a high load, extremely busy content streaming server ? The server fetches the content from remote servers like amazon, s3, etc. then uses php to dynamically stream the content to user without saving it onto the hard drive. php uses CURL to fetch the file, then uses flush() to stream it simultaneously, so not much hard drive work... only network and bandwidth.
The server is quad core xeon, with 1Gbit full duplex NIC, 8gb RAM, and 500GBx2 in RAID. Server memory usage and cpu load is pretty low.
We're running debian lenny and lighttpd2 on it (yes I know its not released yet :-) ) with php 5.3.6 and php fastcgi with spawn-fcgi bind on 4 different unix sockets with 20 children each. Max fcgi requests is 20, with mod_balancer module in lighttpd2 configuration to balance the fastcgi requests among these 4 sockets in SQF (short queue first) configuration.
Our servers use a lot of bandwidth i.e network connection is busy all the time. Just after 100 to 200 parallel connections, the server starts to slow down and eventually becomes unresponsive, starts giving connection timeout errors. When we had cpanel, we never got timeout errors, so it cannot be a script issue. It must be a network configuration issue.
lighttpd2 configuration: worker processes = 8, keep alive requests is 32, keep alive idle timeout is 10 seconds, and max connections is 8192.
Our current sysctl.conf contents are:
net.ipv4.tcp_fin_timeout = 1
net.ipv4.tcp_tw_recycle = 1
# Increase maximum amount of memory allocated to shm
kernel.shmmax = 1073741824
# This will increase the amount of memory available for socket input/output queues
net.ipv4.tcp_rmem = 4096 25165824 25165824
net.core.rmem_max = 25165824
net.core.rmem_default = 25165824
net.ipv4.tcp_wmem = 4096 65536 25165824
net.core.wmem_max = 25165824
net.core.wmem_default = 65536
net.core.optmem_max = 25165824
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_max_orphans = 262144
net.ipv4.tcp_max_syn_backlog = 262144
net.ipv4.tcp_synack_retries = 2
net.ipv4.tcp_syn_retries = 2
# you shouldn't be using conntrack on a heavily loaded server anyway, but these are
# suitably high for our uses, insuring that if conntrack gets turned on, the box doesn't die
# net.ipv4.netfilter.ip_conntrack_max = 1048576
# net.nf_conntrack_max = 1048576
# For Large File Hosting Servers
net.core.wmem_max = 1048576
net.ipv4.tcp_wmem = 4096 87380 524288