Permanent IP block with iptables after first connection to port

Using recent

recent is an abnormal match: it doesn't just filter, it's also doing changes (which is usually done in a target rather than a match): changing entries in a list. It must be used twice like you already did elsewhere: once to set the list, and once anyway to check the list. The second time shouldn't check anything else, so any connection from the IP to anything will match and be dropped, not just to port 22.

Your user chain SSH should be called from INPUT (or FORWARD) before any generic stateful rule like -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT is used.

So replace the start of your chain:

iptables -N SSH
iptables -A SSH -p tcp -m tcp --dport 22 -m state --state NEW -m recent --name banip --rsource --set -j DROP # -j DROP could be left: the next rule would do it.
iptables -A SSH -m recent --name banip --rsource --update -j DROP

You can then add your additional rules for port 987. As they don't state a list name, they'll be using DEFAULT which won't interfere with banip.

Should you want those IP addresses to expire after some time (eg 12345 s), you'd also add for example --seconds 12345 --reap after the --update parameter as documented.

By default the recent's list can accept 100 entries, so to use recent as a ban list, the xt_recent kernel module should be loaded, before loading iptables rules, with an option like ip_list_tot=65535 because a list that can have only 100 entries is probably not large enough to implement a ban list. This parameter is global for all list sizes.

You should also consider using ipset instead, as described below, because it's usually more flexible.

Using ipset

After some level of complexity is reached, better use the companion ipset framework. ipset can handle arbitrary large hashed lists of various network-related types (IP address, (TCP and/or UDP) port...).

You can do this with iptables's set match and SET target + ipset. ipset will act as memory.

Create the set with the companion tool ipset:

ipset create banip hash:ip

This will create a hashed set with 65536 entries by default, which is way better than 100 or 255. See the manual to see how to create a larger set if needed (the set's size once determined can't be increased from the packet path, only by further using the ipset command). You can pre-populate, save, restore, list contents of, etc. this set with appropriate ipset commands.

The rules above can be replaced with:

iptables -N SSH
iptables -A SSH -p tcp -m tcp --dport 22 -m state --state NEW -j SET --add-set banip src
iptables -A SSH -m set --match-set banip src -j DROP

If you want the entries to expire the easiest is to have created the set with a timeout parameter.

One of the methods for fail2ban is to use ipset (but the trigger is from log files rather than packet path).

There are yet other methods possible, like using nftables (and its built-in sets) instead of iptables + ipset. Feel free to test it.

IPtables is not able to manage that kind of dynamic analysis by itself. From your description, what you need is actually a tool that manages these kind of rules for you. Overall, I can recommend fail2ban and denyhosts for that specific case, but there might be others.

I successfully tried both with a similar use-case. You can setup different rules for different ports and protocols, rule expiration, max retries and many other options.

Official links:

http://denyhosts.sourceforge.net/

https://www.fail2ban.org/wiki/index.php/Main_Page

I agree with @J A's answer.

I did however want to share my open-source XDP Firewall project here that should be able to achieve this. The tool utilizes XDP and should be able to block traffic a lot faster than IPTables if your NIC supports XDP-native. Otherwise, it should be the same speed as IPTables if you're using XDP-generic/SKB mode (most NICs/modern kernels should support this).

The only cons to using this program are:

1. It uses a config file to specify filtering rules. There are no commands at the moment to add/remove filtering rules.

2. After the program is restarted, the IP blacklist will be wiped.

I'm hoping to implement commands that'll add filter rules and save them automatically in the future along with the ability to permanently block IPs that support restarts as well.

For your case, the following config should do:

interface = "changeme";
updatetime = 15;

filters = (
    {
        enabled = true,
        action = 0,

        tcpopts = (
            {
                enabled = true,
                dport = 22,
                blocktime = 9999999999
            }
        )
    }
);

This will block packets targeted against destination port 22 over TCP for 9999999999 seconds (you may increase this if need to be since the blocktime config option is a uint64_t type).

I hope this helps in your case!