I got an reoccuring issue with a few remote backup machines I got. I have a few cron jobs that runs a backup schedule but after a while they stop working. I quickly found that even simple pings wont work; I get:
ping: sendmsg: Operation not permitted
This from the main server to a backup host. Ping from a backup host to the main server simply does nothing really. 100% package loss.
The first times this occured I resolved it my getting new IP's or rerouting through another network inteface, but after a while the same issue founds it's way back.
Once I did a flush of the iptables, and that did the trick - but since I locked myself out that time and had to physically connect to the server and add new routing I'd rather not do that again. Rather I would like to find out why this is happening and looking for a more suitable and sustainable resolution.
I'm using a few security measures such as denyhosts and fail2ban, which I guess could be accounted for this issues. But where do I start to look?
Neither "iptables -L OUTPUT" nor "iptables -L INPUT" presents any detail in denied hostnames.
Finally I found the answer. It was - for although unkown reason - PSAD that was the issue (for doing it's job). PSAD denies my machines and running a simple
psad -Fresolved the issue. I have used ufw to allow the machine's IPs and will now see whether the issue will return or not.