I have been using my own self-made "iptables rules" for blocking all the major type of DDOS attacks on game-servers because these attacks were application/game-server specific instead of the general DDOS attacks. It took me almost 3 years to study those attacks and respond with these rules. Up til now, I was using a rate-limit on iptables that would automatically DROP the IP if it made 250 hitcounts/sec as the client rate is much lower than this (maximum it goes is like 500kb/s). This is because attackers would generally send a large number of hit-counts with packets of the length 15:30 as this is the general packet-length where the game-server responds well. This worked really well until someone attacked with something different today to get through this rule.
-A INPUT -p udp -m udp --dport 16000:29000 -m recent --set
-A INPUT -p udp -m udp --dport 16000:29000 -m recent --update --seconds 1 --hitcount 250 -j DROP
Recently, I observed attacks that were easily making 1-3mb/sec of input, as seen through "iftop" to my servers and the above rules weren't blocking them because my game-server was responding with "....disconnect" packets to all the incoming connections. This happens when game-server doesn't recognize the input string or length of the packet. This was the content of the packet:
http://paste.ubuntu.com/6000381/
Now as you might as well see that the packet had a huge length. According to Wireshark where I grabbed the packet, the length of the packet was 700 and data size/length was about 5000 bytes. And I've seen this before that if you have a higher data-length/size of the packet, you can still have a higher rate even with a fewer hitcounts. So this explains maybe why it had an input of around 3mb/s and it didn't get blocked through the hitcounts, because it never had hitcounts greater than 250..
Now I need something that would more be bandwidth specific than the number of packets/second. I require an iptables rule that would automatically block an input>1mb/s and it may use the length of the packet if needed. The general data-size/length of a fair game-server client would usually have packet lengths and data-size of less than 500 bytes but it might go higher in some cases but will never take more than 500kb/s of bandwidth.
The question is different than the rest of previously answered questions because it's firstly application specific. Secondly, I am asking an iptables solution that would deny any input >1mb/s?
Update:
I did some theoretical working to explain why a 3 Mb/s input bandwidth is not more than 250 hitcounts/second and here is the working:
3 Mb = 3x1024x1024 = 3,145,728 bits/sec = 393,216 Bytes/sec
Hence,
If a packet had the size of 5000 bytes so the maximum number of packets per second should be:
393216/5000 = 78.6 Packets/second
And this means that if the iptables rules had to work, the packet should have been smaller than 393216/250 = 1516 bytes.
Have you looked at fail2ban? It may help you react to incoming attacks by dropping the source IPs at the firewall.