My access.log are full of:
Mozilla/5.0 (compatible; DotBot/1.1; http://www.opensiteexplorer.org/dotbot, [email protected])
CCBot/2.0 (http://commoncrawl.org/faq/)
Mozilla/5.0 (compatible; spbot/5.0.3; +http://OpenLinkProfiler.org/bot )
Mozilla/5.0 (compatible; MJ12bot/v1.4.8; http://mj12bot.com/)
Mozilla/5.0 (compatible; SeznamBot/3.2; +http://napoveda.seznam.cz/en/seznambot-intro/)
and of:
85.55.237.132 - - [16/Jul/2018:06:30:44 +0000] "GET /wp-login.php HTTP/1.1" 404 11462 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.1"
51.253.65.140 - - [16/Jul/2018:06:30:47 +0000] "GET /wp-login.php HTTP/1.1" 404 7645 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.1"
113.161.75.37 - - [16/Jul/2018:06:30:57 +0000] "GET /wp-login.php HTTP/1.1" 404 12145 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.1"
114.94.208.2 - - [16/Jul/2018:06:31:04 +0000] "GET /wp-login.php HTTP/1.1" 404 11096 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.1"
112.134.172.163 - - [16/Jul/2018:06:31:07 +0000] "GET /wp-login.php HTTP/1.1" 404 11177 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.1"
144.48.75.34 - - [16/Jul/2018:06:31:13 +0000] "GET /wp-login.php HTTP/1.1" 404 7736 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.1"
And none of my sites is wordpress.
Currently I block access to these via UFW blocking the IPs, but they have become thousands and every day the number increases.
Is there any way to say to Ubuntu for example:
If the user-agent (or requested URL) contains "Mjbot" block it.
I've seen several tools like Fail2ban, modsecurity, etc, but they seem to complicate things rather than simplify them.
I'm looking for a really simple solution.
This can be done easily by the use of
mod-rewrite
. Just modify the start of your.htaccess
to sayYou can replace
*Mjbot*
with a regex of your choiceThe
RewriteCond
directive sets up a condition for the url to be 'rewritten' or modified. In this case, it checks if the variableHTTP_USER_AGENT
matches the regular expression*Mjbot*
with no case sensitivity[NC]
The
RewriteRule
directive tells apache to respond to all requests under this.htaccess
(.*
) to be responded with a 403 Forbidden (theF
flag) and to stop the rewriting process immediately after (theL
flag)Furthermore you can also prevent such requests from cluttering your access log by use of the already packaged
mod_setenvif
To do that, to your
.htaccess
add this line:I think it's not working...
I have this rules in apache2.conf/httpd.conf
If I text a URL with curl, I see this:
Shouldn't I see 403?
In fact if I add the same rule directly on the .htaccess of the site I see this:
So I did a little bit of research on the URL rewrite rules in apache2.conf/httpd.conf, and found lots of confusing people and a lot of confusion.
Among the many things some create a bit of clarity but also other confusion, such as this: http://tltech.com/info/rewriterule-in-htaccess-vs-httpd-conf/
Here it is said that the rules of url rewrite must be enclosed between tags:
But wanting to use apache2.conf/httpd.conf I did not understand in which, there are already 3 in my apache2.conf:
I tried to add a tag:
around my code at the beginning of the file, Apache didn't like it.
So I tried so:
apache does not complain, but it does not seem to work.
So I tried so:
again apache does not complain, but it does not seem to change anything.
Then I tried, not to add a tag, but enter the URL rewrite rules directly in the existing tags, but apache did not like.
Then I searched again and found this: https://stackoverflow.com/questions/344588/how-can-i-implement-a-global-rewritecond-rewriterule-in-apache-that-applies-to
Here they talk about new options for url rewrite: InheritDown, InheritDownBefore, IgnoreInherit.
But again, wanting to use apache2.conf/httpd.conf, where should I put these new options and where my rules?
I'm sure all this can work, but I need to figure out where every single piece has to go to make everything work properly.
Can someone help me please?
Maybe you should buy a book about fail2ban, if there is one.
You posted some more questions in an answer. Wouldn't it make more sense to work on your initial question?
If you want to know my answer/opinion: Change from Apache to Nginx and do something like that:
In error.php you can get more data from the requester via $_SERVER and mail it to you or better put it in a log file.
It is even more fun to prohibit SQL in GET parameters:
Of course, you still need the following.