Is there any standard way to prevent shared web servers from being abused? I run a CPanel box with a few people on it, and I get the occasional person that decides to use it to DoS other websites. My current 'detection' involves looking at Munin graphs for traffic spikes, then poking around on the machine until I find the cause.
Is there any software out there to detect attacks as they happen?
I run the following: LMD - http://www.rfxn.com/projects/linux-malware-detect/ Clanscan
These do an okay job of detecting your standard C99 shell, or commonly used DoS scripts, but it doesn't detect any sort of custom stuff.
I think you're looking for an Intrusion Detection System (IDS), or maybe an Intrusion Prevention System (IPS). Have you looked at Snort?
CSF/LFD (http://www.configserver.com/cp/csf.html) are fantastic (and free!) and work great with cPanel boxes. It also works just fine on non-cPanel boxes.
Takes less than 5 minutes to setup on your existing box. Tweak the defaults for your environment and it'll automatically block IPs and/or kill processes that go beyond the bounds you set, and send you an email of what it did.