How to use iptables or tc to limit packets per client.

I have done this using a combination of TC and iptables hashlimit. I created a TC bandwidth limiter outbound on the LAN interface (to target download traffic) set to 5 Mbits/second. I then use the iptables hashlimit module in the interface's output mangle chain such that if the packet rate exceeds a certain threshold, between any two distinct source and destination IP addresses, it starts to classify those packets into the TC 5 Mbit/s traffic shaping class.

You have to set the packet threshold correctly, though, and I based mine on the fact that my MTU is 1500 bytes so used this to calculate how many packets per second that would be to create a 1.5 Mbits per second threshold (1000 packets/second). Coupled with setting the highest possible burst value on the hashlimit iptables module (which on my setup, appears to be 10,000), the result of all this is that short downloads run at full speed, but longer ones begin to slow down as some packets are passed into the TC bandwidth limiting class, which then obviously lowers the packet rate, etc etc. It is a bit of a hack, but amazingly it does work, and it works well. And because TC is used, no packets are ever dropped - simply delayed and slowed down due to being shoved into the TC class once the packet rate is reached.

This is on a CentOS 6 box. I think the more modern stuff allows for the hashlimit module to support bytes per second, not just packets per second, which is even better, but I tried this on my setup and it just reverts back to using packets per second.

I'm on a mobile phone just now so can't paste any config but if you want some examples, let me know and I'll edit this answer. I really like this solution because the limiting I've gone for is based on source and destination IP. So the system sees each distinct src+dst IP combination as its own packet stream for limiting. The hashlimit module supports source and destination ports as well (basically any combination of source IP, source port, destination IP, destination port), so you could even do the limiting on a per session basis between source IP+port and destination IP+port.

UPDATE

So below is roughly how you do it; experimentation will be required.

tc:

/sbin/tc qdisc add dev eth0 root handle 1: htb
/sbin/tc class add dev eth0 parent 1: classid 1:1 htb rate 5Mbit

iptables:

#!/bin/bash

[[ "$1" =~ ^[ADI]$ ]] || exit 1

for IPT in /sbin/ip{,6}tables
do
  $IPT -t mangle -$1 POSTROUTING -o eth0 -m hashlimit --hashlimit-above 1000/second --hashlimit-burst 10000 --hashlimit-mode srcip,dstip --hashlimit-name limiter -j CLASSIFY --set-class 1:1
done

you can solve this problem in a simple way, try to use recent module with iptables, recent keep track of the source address:

iptables -m recent -h
recent match options:
[!] --set                       Add source address to list, always matches.
[!] --rcheck                    Match if source address in list.
[!] --update                    Match if source address in list, also update last-seen time.
[!] --remove                    Match if source address in list, also removes that address from list.
    --seconds seconds           For check and update commands above.
                                Specifies that the match will only occur if source address last seen within
                                the last 'seconds' seconds.
    --reap                      Purge entries older then 'seconds'.
                                Can only be used in conjunction with the seconds option.
    --hitcount hits             For check and update commands above.
                                Specifies that the match will only occur if source address seen hits times.
                                May be used in conjunction with the seconds option.
    --rttl                      For check and update commands above.
                                Specifies that the match will only occur if the source address and the TTL
                                match between this packet and the one which was set.
                                Useful if you have problems with people spoofing their source address in order
                                to DoS you via this module.
    --name name                 Name of the recent list to be used.  DEFAULT used if none given.
    --rsource                   Match/Save the source address of each packet in the recent list table (default).
    --rdest                     Match/Save the destination address of each packet in the recent list table.
    --mask netmask              Netmask that will be applied to this recent list.

example for blocking ssh brute force:

iptables -A INPUT -i eth0 -p tcp --syn --dport 22 -m recent --name ssh --set
iptables -A INPUT -i eth0 -p tcp --syn --dport 22 -m recent --name ssh --rcheck --seconds  30 --hitcount 2 -j DROP

You might be able to solve this using the connlimit module:

http://www.cyberciti.biz/faq/iptables-connection-limits-howto/