Today I've been dealing with a server suffering from what looked like a SYN flood attack. It was a bit of a rush to get the site back online, so we did these three steps to bring the service back to a usable state. The server load was low during the attack, so it didn't bring the server down, it just timed out HTTP visitors.
Now I don't believe these solved the issue, but they certainly resolved the symptons until the flood subsided.
Set sysctl -w net.ipv4.netfilter.ip_conntrack_tcp_timeout_syn_recv=5
Increased Apache prefork ServerLimit and MaxClient to 512 (from 256).
Set Apache ListenBackLog to 1024
I saw various iptables --limit options being discussed elsewhere on the web, however we concluded that these would limit legitimate traffic, as each item of the webpage being requested (each image etc) would count towards this limit, stopping the page from loading fully.
What do people do in these situations, and were our action wise since load wasn't an issue?
I would use a firewall at the network perimeter to prevent\remediate SYN flood attacks (as well as DOS, DDOS, spoofing, port probes, address space probes, etc.). I don't want this type of stuff getting into my internal network, where I'll have to deal with it on a machine by machine basis.
Since I am not an expert in iptables, I generally let one of two firewalls handle this for me. Both APF and CSF are great firewalls when it comes to protection from SYN attacks, as well as a multitude of other ways folks can attack your server.
I do not know your specific configuration, but I have used both of the said firewalls on "general" cPanel/DirectAdmin/Plesk servers as well as some with custom services and it works great once you allow the right ports.
Separately, you may wish to turn on SYN Cookies, which helps mitigate attacks where SYN is left open. Both of the above scripts have this as an option.