Recently my FreeBSD 8.0 (GENERIC) box was hit with a large amount of requests from an IP in Taiwan, trying to guess passwords and all of that stuff. Anyway, long story short, I noticed at a certain point I couldn't ssh into the box. After logging in directly, I noticed a huge number of password guesses, and the message msk0: watchdog timeout
. msk0 referring to my wired ethernet connection.
I brought the interface back up with ifconfig msk0 up
, and was successfully able to ping the address of that interface. However, when trying to ping my main router, which the box is directly connected to, it hung. Attempting to ping my external IP address returned a whole bunch of sendto: no buffer space available
.
The problem was resolved with a reboot, but obviously this isn't the ideal way to go about it. In the case of something like this happening again, what steps should I take to restore connectivity? I've read that sometimes it can be prevented with watchdog -t 0
, but I'm not sure I want to go down that avenue.
When it comes to prophylaxis, is there a way to refuse connections from IPs that have a certain number of failed logins for a certain period of time? For example, 15 failed logons would result in refused connections for the next twelve hours?
I missed the last part of your question with my other answer, so I'll add it here quick.
I use and highly recommend to everyone with a public facing *nix server: Fail2Ban
It's in the ports tree under security/py-fail2ban, so it's easy to get started.
After it's built open
/usr/local/etc/fail2ban/jails.local
in your favorite editor. Here's a quick start if you use IPFW. If you use pf it would be slightly different.Enable the service
echo 'fail2ban_enable="YES"' >> /etc/rc.conf
and start it/usr/local/etc/rc.d/fail2ban start
Monitor the contents of table 1 in IPFW for a while to be sure you're not going to lock yourself out with
ipfw table 1 list
. Once it's working as expected add a firewall rule to block IPs in table 1:ipfw add 00030 deny ip from "table(1)" to me
. Be sure to add this to your start-up ruleset so it's loaded on reboot.Fail2Ban can be used to monitor just about any log file. Most services already log failed logins to a log file, and most others can be made to; and there's plenty of examples found via Google too (or asking here).
I don't know about chips that msk0 supports; but I've seen similar problems on a lot of other cards/chips/drivers. 99.9% of the time it's a buggy firmware implementation (usually due cheap manufacturing) not handling the watchdog timer(s) correctly.
Additionally, that's a Marvel chip, and Marvel has not been open source friendly; it could be a problem with the driver itself. Either way the best place to start is the bottom of the Handbook page on Troubleshooting NICs.
I've had my fair share of these sorts of problem; I've found the easiest solution is to switch to more expensive NICs (though you can find older ones on eBay for next to nothing; my first choice for home equipment).
If the troubleshooting doesn't resolve the issue, there's more troubleshooting experts on the FreeBSD forums too.