Using mod_rewrite I can construct a rule to respond with a clean error code (e.g. 404 not found, 410 gone, or 403 unauthorised) when a page is requested that I don't want to serve.
But frequently I get completely erroneous requests from hackers scanning my website for vulnerabilities or possibly cross-site scripting attempts. For these customers I do not want to return a clean error - I'd rather do something else like immediately drop the connection with no response or, alternatively, hold the connection open for a lengthy period of time to frustrate the automated process.
Any ideas how to accomplish this with Apache? I've read that nginx has the ability to immediately terminate a connection when a particular pattern is matched.
Is it really worth the trouble? As long as your site doesn't have any vulnerabilities that the bots are looking for, it's just another entry in your error log. Unless it's actually causing excessive load on the server, it's pretty much a non-issue.
If you are running something popular in the sense of wide-spread usage and vulnerability exploits and you aren't confident about staying up to date on security fixes, then, yes, mod_security might be a layer of safety. Note, however, that mod_security will break stuff with any useful rules unless it's tuned properly for your site(s).
I don't think mod_security will give you a tar-pit or null response like you are asking for. If you want to tell some(one|thing) to piss off, give them a 403 Forbidden on any specific URL's you are especially worried about by setting up the appropriate rewrite rules.
You may want to consider
mod_security.http://www.modsecurity.org/