How can I set login throttling and lockout policies on a Linux (specifically Debian) system?
Things like, setting retry count, delays between login attempts, and idle session timeouts.
The box in question is a headless server, so SSH settings are acceptable. Thanks!
On my DMZ box...
I primarily use a combination of DenyHosts and SSHD configuration tweaks to accomplish those things, which are obviously heavily dependent on your particular goals. For security reasons, I'm not inclined to post my denyhosts.conf file - but the FAQ (linked herein) should hit on a few of your questions quickly.
Matt's recommendations are sensible - but you should also have a look at how pam is configured.
Note that this is not the way to prevent completely unauthorized system access - for that you need to have a look at the sshd config, firewall, fail2ban, and possibly 2-factor authentication.