Best way to use large IP blacklist to deny access to a web server?

You could try using moblock (google it - can't add links yet, new user). Disable all downloaded blocklists, and use only a local blocklist that you generate. You may need to add NFQUEUE (netlink queue) support to your kernel, but it may already be there by default.

The general setup is: for all SYN packets on the ports you want to filter, use netfilter's NFQUEUE action to push them to moblock, sitting in userspace. Moblock does efficient matching and sends back either an ACCEPT or DROP response to netlink.

The moblock config file format is quite simple: on each line, give a name and an IP range, in the form 123.123.123.42-123.123.124.56. When moblock loads the ranges, it builds an efficient data structure to match against those ranges. When a packet is dropped because of a match, the range name and actual source IP is logged (or not, if you disable logging of matches).

I've used moblock in its default configuration (downloaded blocklists) with about 230000 IP ranges, and have observed no discernible performance hit (filtering only the SYN packet is important to keep kernel/userspace traffic down though).

One caveat: if moblock is not running, I believe NFQUEUE's default action is to DROP, resulting in a denial of service of your application. That said, I've had moblock running continuously without any problem for over 6 months. Still, you may want to set up a monitoring probe that alerts you if a known-good IP can no longer connect to :80 on your server. You definitely don't want to use moblock to filter ssh, unless you have explicitely whitelisted some trusted IPs in netfilter to recover.

AFAIK Deny rules in Apache config and iptables have linear lookup time.

Sort of. You can use chains in IPTables to break it up, i.e. a simplistic approach would be to have a chain for each A class block of addresses (e.g. 1.0.0.0/8, 2.0.0.0/8, etc.) and the rules for that block within the chain, reducing the look up time significantly (i.e. worst case ~200 rule evaluations to get to the A class rule and then however many rules within that block. You can also use "iptables -L -v -n" to see which rule sets are being evaluated the most heavily and move them to the top. There are other better ways to do this, iptables documentation covers this.

Yeah, on your system, there should be a /etc/hosts.allow. Take a look, it has some very easy examples to lock out IPs even on a per service basis.

In a nutshell, in /etc/hosts.allow:

ALL : ALL : allow

httpd : /etc/hosts.httpd.deny : deny

sshd : /etc/hosts.sshd.deny : deny

I think you cannot do it without paching you kernel.

ipset seems to be a good solution.

Quoting from its webpage:

If you want to

• store multiple IP addresses or port numbers and match against the collection by iptables at one swoop;
• dynamically update iptables rules against IP addresses or ports without performance penalty;
• express complex IP address and ports based rulesets with one single iptables rule and benefit from the speed of IP sets

then ipset may be the proper tool for you.

You can also do it in .htaccess

Here's an example which blocks by useragent, but the idea is the same: http://jetfar.com/trap-content-scraper-spam-harvester-bots-using-honeypot-wordpress-htaccess/

AFAIK Deny rules in Apache config and iptables have linear lookup time.

I would try it and then do some testing before you rule this out - the impact might be unnoticeable, and it would be very easy to maintain.

Similarly you could try adding the list to /etc/hosts.deny

I wouldn't assume which is faster without testing both first.

Cheers / Richy