I have over 2.5 million ssh attempts and fail2ban and only a few thousand ip's blocked. The find time was set to 600, max retries 4, and bantime -1. The server is physically present. I changed the jail.local and tried restarting systemctl restart fail2ban and it starts with a whole bunch of errors. I reverted jail.local back how it was and tried again and still get the same errors. I deleted some of the secure logs and tried again and it seemed to start but when I reviewed the status the errors were still there and almost nothing is being blocked.
I get the error Fail2ban ipset create fail2ban-sshd hash: ip timeout -l followed by a whole bunch of subsequent failures that I think are precipitated by the first.
I tried stopping, uninstalling and reinstalling fail2ban and still have the same initial error followed by a whole bunch of errors.
My questions are:
1. Why/how does ipset create fail2ban-sshd hash timeout?
2. How can I properly reinstall fail2ban so that maybe it might block something?
3. If there's no working answer on the first two, can't the ip blocking rules be programmed outside of fail2ban?
fail2ban.log:
2019-05-12 21:08:35,823 fail2ban.action [1730]: ERROR ipset create fail2ban-sshd hash:ip timeout -1
firewall-cmd --direct --add-rule ipv4 filter INPUT 0 -p tcp -m multiport --dports ssh -m set --match-set fail2ban-sshd src -j REJECT --reject-with icmp-port-unreachable -- stdout: ''
2019-05-12 21:08:35,823 fail2ban.action [1730]: ERROR ipset create fail2ban-sshd hash:ip timeout -1
firewall-cmd --direct --add-rule ipv4 filter INPUT 0 -p tcp -m multiport --dports ssh -m set --match-set fail2ban-sshd src -j REJECT --reject-with icmp-port-unreachable -- stderr: "ipset v6.29: Syntax error: '-1' is out of range 0-4294967\n\x1b[91mError: COMMAND_FAILED\x1b[00m\n"
2019-05-12 21:08:35,824 fail2ban.action [1730]: ERROR ipset create fail2ban-sshd hash:ip timeout -1
firewall-cmd --direct --add-rule ipv4 filter INPUT 0 -p tcp -m multiport --dports ssh -m set --match-set fail2ban-sshd src -j REJECT --reject-with icmp-port-unreachable -- returned 13
2019-05-12 21:08:35,824 fail2ban.actions [1730]: ERROR Failed to start jail 'sshd' action 'firewallcmd-ipset': Error starting action
2019-05-12 21:08:35,825 fail2ban.actions [1730]: NOTICE [sshd] Ban 218.92.0.147
2019-05-12 21:08:46,771 fail2ban.transmitter [1730]: WARNING Command ['start', 'sshd'] has failed. Received OperationalError('database is locked',)
2019-05-12 21:08:51,787 fail2ban [1730]: CRITICAL Unhandled exception in Fail2Ban:
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/fail2ban/server/jailthread.py", line 66, in run_with_except_hook
run(*args, **kwargs)
File "/usr/lib/python2.7/site-packages/fail2ban/server/filtersystemd.py", line 290, in run
self.jail.putFailTicket(ticket)
File "/usr/lib/python2.7/site-packages/fail2ban/server/jail.py", line 195, in putFailTicket
self.database.addBan(self, ticket)
File "/usr/lib/python2.7/site-packages/fail2ban/server/database.py", line 96, in wrapper
return f(self, self._db.cursor(), *args, **kwargs)
OperationalError: database is locked
2019-05-12 21:08:56,800 fail2ban.actions [1730]: ERROR Failed to get all bans merged, jail 'sshd': database is locked
2019-05-12 21:09:01,812 fail2ban.actions [1730]: ERROR Failed to get jail bans merged, jail 'sshd': database is locked
2019-05-12 21:09:01,941 fail2ban.action [1730]: ERROR ipset add fail2ban-sshd 218.92.0.147 timeout -1 -exist -- stdout: ''
2019-05-12 21:09:01,941 fail2ban.action [1730]: ERROR ipset add fail2ban-sshd 218.92.0.147 timeout -1 -exist -- stderr: 'ipset v6.29: The set with the given name does not exist\n'
2019-05-12 21:09:01,942 fail2ban.action [1730]: ERROR ipset add fail2ban-sshd 218.92.0.147 timeout -1 -exist -- returned 1
systemctl status fail2ban
fail2ban.service - Fail2Ban Service
Loaded: loaded (/usr/lib/systemd/system/fail2ban.service; enabled; vendor preset: disabled)
Active: active (running) since Sun 2019-05-12 21:08:46 EDT; 38min ago
Docs: man:fail2ban(1)
Process: 1462 ExecStop=/usr/bin/fail2ban-client stop (code=exited, status=255)
Process: 1726 ExecStart=/usr/bin/fail2ban-client -x start (code=exited, status=0/SUCCESS)
Main PID: 1730 (fail2ban-server)
CGroup: /system.slice/fail2ban.service
├─1599 /usr/bin/python2 -s /usr/bin/fail2ban-server -s /var/run/fail2ban/fail2ban.sock -p /var/run/fail2ban/fail2ban.pid -x -b
└─1730 /usr/bin/python2 -s /usr/bin/fail2ban-server -s /var/run/fail2ban/fail2ban.sock -p /var/run/fail2ban/fail2ban.pid -x -b
May 12 21:08:24 myHost systemd[1]: Starting Fail2Ban Service...
May 12 21:08:24 myHost fail2ban-client[1726]: 2019-05-12 21:08:24,579 fail2ban.server [1728]: INFO Starting Fail2ban v0.9.7
May 12 21:08:24 myHost fail2ban-client[1726]: 2019-05-12 21:08:24,580 fail2ban.server [1728]: INFO Starting in daemon mode
May 12 21:08:29 myHost fail2ban-client[1726]: ERROR NOK: ('database is locked',)
May 12 21:08:34 myHost fail2ban-client[1726]: ERROR NOK: ('database is locked',)
May 12 21:08:46 myHost fail2ban-client[1726]: ERROR NOK: ('database is locked',)
May 12 21:08:46 myHost systemd[1]: Started Fail2Ban Service.
[root@myHost /]# fail2ban-client status sshd
Status for the jail: sshd
|- Filter
| |- Currently failed: 7
| |- Total failed: 46
| `- Journal matches: _SYSTEMD_UNIT=sshd.service + _COMM=sshd
`- Actions
|- Currently banned: 1
|- Total banned: 1
`- Banned IP list: 218.92.0.147
After reviewing the logs over and over I decided to pay attention to the following line:
This got me to check the
firewalldlog which showedfor each time I tried starting and restarting fail2ban. I searched that error and found https://www.centos.org/forums/viewtopic.php?t=60586 which suggested changing adding
to the jails.local file under the sshd section.
I restarted fail2ban and the ip's started getting blocked in a way that was more consistent with my expectations.