I run a website which is being abused every day with bots trying to send spam, not respecting robots.txt and so on. The biggest problem, however is repeating attempts to massively harvest website. Right now i have my own script which uses Nginx to block misbehaving ips. This list contains like 100000+ ips and it's still growing.
Now the problem: I am migrating my website to a new server. I'd like to ask for advice on how to block such amount of ips. The most popular solution is Fail2Ban, however it is able to ban using iptables which have their own limitations (around 4500 rules, see How many rules can iptables support?) So, blocking maximum of 4500 ips is definitely not enough for me.
I'd like to know what is the best way to overcome this limitation.
Should i add rules to fail2ban to block ips using nginx or should i use another approach ?
Anyway, i got it working with Fail2Ban by installing Ipset and changing the default Fail2Ban action to use ipset for banning ips.
I would suggest looking into mod_security (https://www.modsecurity.org/). It can rate limit requests, which is probably what you're looking for.