I want to dynamically block specific connections that use the same IP address based on a rate or connection limit. Is this possible using Solaris/IPF or some sendmail extension? I want to limit sendmail login attempts to prevent brute force attacks.
In Linux it's easily handled on the iptables firewall layer, but I haven't been able to figure out a way to use ipf to limit it on the firewall layer. Sendmail has a built-in rate limit and connection limit, but it appears to be applied to all users so if we're experiencing a DOS or DDOS it would block all our users instead of just the attacker.
take a look here, these are sendmail based directive which can prevent flooding and bad behaviours, maybe this helps you: http://www.acme.com/mail_filtering/sendmail_config.html
I for myself use these configs:
Further you can search for an Implementation called greypit. I'm not really up to date on that topic, but greypit should have ip base connection limits, maybe theres a solaris version out there.
Another way is as follow. Check your logs for massive dos activities or false logins and use the greetpause in access. If you identity malicious behaviour insert a line as follows in your access and regenerate your access.db
From now each request from the ip or hostname hast to wait 100 seconds before getting a helo.
I used this feature the other way around, but it can also be used for blocking unwanted connections.
The script which did these entrys was just a cron script, but care it is just the other way around getting good traffic and you have to manually recreate your access.db: