Home / server / Questions / 305145
In Process
Chris
Asked: 2011-08-26 20:12:15 +0800 CST

How to configure a redhat/centos/amazon linux server for 1million open tcp connections?

  • 772

I've been reading the following article:

And wondering if there's anything else i need to know about tuning linux to handle 1million tcp connections? So far i've narrowed it down to the following:

  • Configuring the kernel to support 1mil connections, system wide (sysctl.conf)
  • Configuring to have 1mil connections for the specific user (/etc/security/limits.conf)
  • Configuring tcp stack memory settings (sysctl.conf?)

Is there anything else i need to configure? (this is for an EC2 large 64-bit server)

-edit-

It's not apache, its a libev-based custom coded C server FYI. It'll scale to 1mil just fine, its the kernel that's my worry :)

linux redhat amazon-ec2 tcp

3 Answers

  1. You have most of the tunables configured that I would have set (and had to set). One thing I found when we scaling like this was that you will always have something special to your environment that no one else mentioned. To catch this you need to make sure you are watching and alerting on:

    • errors via syslog
    • errors your program sees like socket() failures, etc
    • network buffer availability (via SNMP or netstat cron)
    • kernel table limits (again via SNMP or /proc file parsing crons)
    • frequent monitoring (very lightweight polls done every 1-10ms, we use OpenNMS which does this really easily, because OpenNMS is awesome).

    One other thing you might run into is issues with the HZ value. On our FreeBSD systems we increased this. I was investigating another question on linux and ran into a case where the socket queues are cleaned in relation to the HZ value:

    TIME_WAIT connections not being cleaned up after timeout period expires

    Regarding the comment I don't think FreeBSD specifically will be any better at this, they both need massive amounts of tuning to work. We are using FreeBSD because the boxes directly connect to the internet and OpenBGPD is currently the best open source BGP implementation available.

    • 2

  2. I was having problems configuring the per-user settings, i kept changing the number of files in the limits.conf but it would then fail to allow me to log in ever again, i had to terminate the ec2 instance. I since discovered that 1024*1024 is the maximum you can put in the limits.conf, any more and you'll bork it:

    echo "* soft nofile 1048576" >> /etc/security/limits.conf 
echo "* hard nofile 1048576" >> /etc/security/limits.conf

    Other than that, and the additions to sysctl that are mentioned in the mochiweb blog links, is there anything else i need to do?

    Thanks a lot guys. I'm really keen to see if i can push this server to handle 1mil connections...

    • 1

  3. It depends on the specs of the server and what type of content it serves. It may not be possible at all, (simultaneous right?) if it's serving active content and media. Database and small websites should do fine. In apache config, try making it start many worker/child threads. In the box itself, just test it to see if Apache will handle it. If apache can but not the box, maybe someone else can help with that. Sorry. Good luck.

    Yes, sorry for not reading it thoroughly.

    • -2