Blocking Invalid HTTP Requests to PHP Application

1. Blocking IP adresses will be a race you can't possibly win. These request usually come from botnets or hacked systems. I would suggest blocking IP just as a temporary solution to a concrete incident where the requests cause problems on you side.

2. I'm not aware of such a list

3. Both will work. But I assume (not tested) that just ignoring the requests will actually be less CPU intense

4. Use a reverse proxy (e.g. varnish or even mod_cache) to cache negative hits (404). So that requests to the same (non existing) URLs can be handled very fast and dont require checking the filesystem everytime.

5. Unaware

1. Will, in general, blocking by IP address work? I know it's been a long time since "IP Address == unique device on Internet", but I'm wondering if these sort of probes generally come from the sort of networks where it'd be safe for me to just block them outright

You can quite easily block many of the requests using a simple .htaccess file. There you can block IPs, urls and plenty of things. But I'm not sure what the source of your "bad requests" are. What I do know is that you should start by stopping the bad traffic that we know of. And this can be done if you make your goal a bit bigger, and rather stop denial of services attacks while at the same time limiting bad requests! Everything you need is at this very useful resource. However they don't really say which modules to install. I recommend: mod_antiloris, mod_evasive but you can find loads more here.

I would personally look at setting up some of those before moving on to hard-blocking certain urls or ips. However, if you want to start limiting specific patterns, it might be easier doing so using a PHP scrict. I.e. Route all paramters to index.php and analyse them there. This would still require a re-route using the .htaccess file. Drupal does something like this:

# Pass all requests not referring directly to files in the filesystem to
# index.php. Clean URLs are handled in drupal_environment_initialize().
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteCond %{REQUEST_URI} !=/favicon.ico
RewriteRule ^ index.php [L]

By doing this, you can "trap" every incominng url. Drupal actually has this built-in and will tell you that person X was looking for file Y. And Drupal again also has modules that can block certain access with certain rules. If that is possible, I'm sure hooking into PHP will expose you to tons of different options you can use to block or not block access from ips.

I think I proposed a solution, but i do need more info to advise further. If you do the above, you will be able to gather more information to perhaps pinpoint the exact source of your bad request woes. Using these tools you will be able to see patterns and at the very least, learn better ways to configure rules to block the bad guys.

2. If I can't block by IP address, does anyone maintain a list of URLs that bad actors generally probe for so I can block by URL?

There are apache modules that does this and make use of their own libraries. There are also libraries for PHP that does this and various networks that keep track of "bad guys", whether spammers using IPs, or spamming using Email addresses etc. Here's an entire list of people keeping track of servers that get blacklisted for a variety of reasons. Try it out by entering www.google.com.

3. Re #2. If I was going to handle this blocking at the server level, which apache module is right for this sort of thing? (MOD_REWRITE, MOD_SECURITY)

MOD_REWRITE would work to get the request to a PHP file, after which you can deal with the problem in PHP. But this does have a bit of overhead. You are better of using MOD_SECURITY and maybe MOD_EVASIVE

4. Is there a better way to block these requests other than by IP or URL?

It really depends. You must study the patterns that emerge and identify the cause. I got very frustrated that we kept getting requests for "transparent.png" (or something) which turned out to be a new standard request for many mobile phones. We thought it was bad, it was good. Don't end up doing that.

5. Also, the system is hosted on EC2 -- does amazon offer any help with this sort of thing?

I don't know. Out of my own personal experience, which was more with the using it to SEND info, we got blacklisted quite quickly when even sending less than 2500 emails. But if you are hosting with them and want them to block incoming "bad requests", they should already be doing that to some extent. Unless you have a mass bot army attacking your server every few days, should you ask them to intervene. Perhaps ask them to help you identify the source, or do your own investigation and decide from there.

In my understanding there's no perfect documented way to deal with such issues. You can only minimize the suspicious attempts on your web application.

Yes, IPs can be blocked on Apache.Check Apache docs here . It uses mod_rewrite and gives 403 response code to the client if IP is in the blacklist. It's a painful job to monitor the logs and maintain hosts.deny

Little less painful job than the first option is if you know the pattern of these invalid URLs, use mod_rewrite to block the requests according to the pattern that was requested. Extending the rule used in a first option, add extra conditions to the rule.

RewriteMap hosts-deny txt:/path/to/hosts.deny
RewriteCond ${hosts-deny:%{REMOTE_ADDR}|NOT-FOUND} !=NOT-FOUND [OR]
RewriteCond ${hosts-deny:%{REMOTE_HOST}|NOT-FOUND} !=NOT-FOUND [OR]
RewriteCond %{QUERY_STRING} <any suspicious keyword/pattern> [NC,OR]
RewriteCond %{REQUEST_URI} <any suspicious keyword/pattern> [NC] 
RewriteRule ^ - [F] 

Add different rules based on what your site does. Say if your site only serves .php files just add one more condition to the above code. RewriteCond %{REQUEST_URI} !.php$ [NC]

Fail2ban is designed to do exactly what you describe. E.g. see http://linuxaria.com/howto/how-to-protect-apache-with-fail2ban

You can configure it to be sensitive to (for example) 404 responses. Or you could even set up honeypots at the URLs to raise immediate black flags for fail2ban to act on.

Regarding specific questions:

1. yes it will work, but beware you don't DOS yourself by having bad links

2. Such a list would be out of date quite quickly - that's why you should use the traffic itself as a datasource. But /^/cgi-bin/ and /.asp(x+)$/ are things you might want to black flag

3. No modules needed unless you want to redirect the black flagged URLs to a more sophisticated handler

4. You're paying Amazon for a service and asking us what you are paying for?