I have a small raspberry Pi that I use as a NAS, mount with fstab, and back up to my desktop with rsync.
But, sometimes the raspberry pi doesn't mount, and rsync deletes the whole backup folder.
Is there any way to tell rsync (or grsync) not to start the backup if the raspberry is not mounted?
I've changed the ssh port (just in case), now I was thinking to close port 22, to block immediately with ufw anyone trying to do damage on my server...
But when i initially enabled ssh i enabled a profile called OpenSSH, and i got the doubt it can do other things too. with this:
sudo ufw allow ssh
For example my vps can be accessed with vnc from provider control panel, if i close port 22 i risk to block vnc too? Or do other damage?
And how should i do it now?
sudo ufw deny ssh
sudo ufw deny OpenSSH
sudo ufw deny 22/tcp
What is the best and fastest way to replicate apache virtual hosts to a new server?
When I need to change server I recreate all domains one by one with a procedure similar to this one:
sudo mkdir -p /var/www/html/sito.com
sudo chown -R $USER:$USER /var/www/html/sito.com
sudo chmod -R 755 /var/www/html/sito.com
sudo nano /etc/apache2/sites-available/sito.com.conf
------
<VirtualHost *:80>
ServerAdmin [email protected]
ServerName sito.com
ServerAlias www.sito.com
DocumentRoot /var/www/html/sito.com
<Directory /var/www/html/sito.com>
Options Indexes FollowSymLinks MultiViews
AllowOverride All
Order allow,deny
allow from all
# New directive needed in Apache 2.4.3:
Require all granted
</Directory>
ErrorLog ${APACHE_LOG_DIR}/error.log
CustomLog ${APACHE_LOG_DIR}/access.log combined
</VirtualHost>
-------
sudo a2ensite sito.com.conf
sudo systemctl reload apache2
and with 50, 100 domains this procedure is slow and tedious ...
Is there a fast and secure way to replicate the virtual hosts of the old server to a new server? Can there be problems with permissions and file owners?
I have ssh access and server root, ubuntu 20.04 server, apache, php8.1
What is the best and fastest way to move mysql when changing servers?
Currently I do it through phpmyadmin, I backup the db in the old server, open a new user in the new server, create a new database and upload/restore the backup.
(I use something like this to speed up the database restore with mysql terminal.)
mysql -u username -p database < /var/www/html/mysql_dump/nomedb.sql
A long and tedious operation... can it be done in a better and faster way to copy database and mysql users to the new server?
I have ssh access and server root, ubuntu 20.04 server, apache, php8.1
----------update---------
Just for completeness, I regularly use this script to make mysql backups, and I download everything with rsync to a local disk, so backing up databases is not a problem. The tedious part is restoring them one by one... https://www.alebalweb-blog.com/66-backup-automatico-mysql-su-server-ubuntu.html (Sorry, the description is in italian, but it's almost all code...)
I've already done some filters for my fail2ban, but just simple things, like:
[Definition]
failregex = ^ .* "GET .*/wp-login.php
ignoreregex =
i don't use wordpress on my server, so i block a lot of malicious attempts. And I have also created similar ones, for: phpmyadmin, wp-admin, wp-include, etc.
but i found in my access.log weird things like:
167.172.145.56 - - [22/Sep/2021:06:44:50 -0700] "GET /wp-login.php HTTP/1.1" 403 9901 "http://cpanel.alebalweb-blog.com/wp-login.php" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:62.0) Gecko/20100101 Firefox/62.0"
167.172.145.56 - - [22/Sep/2021:06:44:50 -0700] "GET /wp-login.php HTTP/1.1" 403 9901 "http://mail.alebalweb-blog.com/wp-login.php" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:62.0) Gecko/20100101 Firefox/62.0"
61.135.15.175 - - [22/Sep/2021:05:45:24 -0700] "GET / HTTP/1.1" 200 26210 "http://webdisk.alebalweb-blog.com/" "Mozilla/5.0 (Linux; Android 10.0; MI 2 Build/O012) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4472.114 Mobile Safari/537.36"
61.135.15.175 - - [22/Sep/2021:05:45:24 -0700] "GET / HTTP/1.1" 200 26210 "http://webmail.alebalweb-blog.com/" "Mozilla/5.0 (Linux; Android 10.0; MI 2 Build/O012) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4472.114 Mobile Safari/537.36"
61.135.15.175 - - [22/Sep/2021:05:45:24 -0700] "GET / HTTP/1.1" 200 26210 "http://cpcalendars.alebalweb-blog.com/" "Mozilla/5.0 (Linux; Android 10.0; MI 2 Build/O012) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4472.114 Mobile Safari/537.36"
Those subdomains don't exist.
I tried to create a new filter, inspired by apache-badbots, but I'm not sure it's correct:
[Definition]
varioustoblock = cpanel\.|store\.|webdisk\.|autodiscover\.|app\.|cpcalendars\.|cpcontacts\.|webmail\.|mail\.|fulaifushi\.|surf11818\.|asg\.|owa\.|exchange\.\$
failregex = ^<HOST> -.*"(GET|POST).*HTTP.*".*(?:%(varioustoblock)s).*"$
ignoreregex =
datepattern = ^[^\[]
especially for the (.), in the past I had problems with the (.) in the fail2ban filters, and the solution was to remove them completely...
but in this case, they can't be removed, I can't block anyone who has the word "mail" in my url... i need to be sure to block "mail."
and I'd like to create a single large filter that identifies both non-existent subdomains and attempts to access wordpress or phpmyadmin, but pyton regex is really scary if you've never used it...
Can anyone help me?
(I also thought about removing * .alebalweb-blog.com from the dns configuration, but I'm not sure it's a good idea, also because I use some subdomains.)
P.s. How worried should I be if someone tries to access subdomains that do not exist on my site?
I monitored the google ip that accessed my server for a few years and identified these masks: 66.249.64.0/19 66.102.0.0/20 64.233.160.0/19 34.64.0.0/10 216.58.192.0/19 74.125.0.0/16
Then I created this script in php which I include in all my websites.
include_once('function_global/cidr_match_function.php');
if(cidr_match($_SERVER['REMOTE_ADDR'], '66.249.64.0/19') === false && cidr_match($_SERVER['REMOTE_ADDR'], '66.102.0.0/20') === false && cidr_match($_SERVER['REMOTE_ADDR'], '64.233.160.0/19') === false && cidr_match($_SERVER['REMOTE_ADDR'], '34.64.0.0/10') === false && cidr_match($_SERVER['REMOTE_ADDR'], '216.58.192.0/19') === false&& cidr_match($_SERVER['REMOTE_ADDR'], '74.125.0.0/16') === false){
$fake_google_ip_list = file_get_contents('function_global/ip_add_fwd.txt');
if(strpos($fake_google_ip_list, $_SERVER['REMOTE_ADDR']) === false){
file_put_contents('function_global/ip_add_fwd.txt', $_SERVER['REMOTE_ADDR'].PHP_EOL , FILE_APPEND | LOCK_EX);
}
}
}
The result is this file containing a list of IPs that pretend to be google and access my server with scam intentions.
Then I, more or less once a day, run this console command which adds all these ip to ufw:
while read line; do sudo ufw insert 1 deny from $line to any; done < /var/www/html/function_global/ip_add_fwd.txt
and upload a new blank file to the server.
I would like to automate this last part and maybe use fail2ban instead of filling more and more of ip, which may never come back, in ufw.
And I think I need a .sh script, or something similar, but I have no idea how to write it... (I am a php programmer and I only use ubuntu as a web server limited to what is necessary for the functioning of my sites).
The script should be:
Can anyone help me write it or at least give me some hints to get started? I have no idea where to start, I tried to put pieces of some other files together, but it didn't go very well ... :(
My phpmyadmin says: A new version of phpMyAdmin is available and you should consider updating it. The most recent version is 4.9.4, released on 2020-01-08.
I followed this guide How to upgrade PHPmyAdmin [revisited], and now it says: Version information: 4.9.1deb2, latest stable version: 4.9.4
What do I have to do to get 4.9.4?
My server is Ubuntu 18.04.
I added a php code to my sites that created errors like this:
AH00484: server reached MaxRequestWorkers setting, consider raising the MaxRequestWorkers setting
and before I figured out what was actually going on, I went to change the values on /etc/apache2/mods-enabled/mpm_event.conf.
Now my values are:
<IfModule mpm_event_module>
StartServers 2
MinSpareThreads 25
MaxSpareThreads 75
ThreadLimit 64
ThreadsPerChild 25
MaxRequestWorkers 400
MaxConnectionsPerChild 10000
</IfModule>
Now I understand and solved the problem, and I would like to restore the default values, just to not make other messes, but I have absolutely no idea what they were.
Can someone tell me the default values, please?
I often have to add new rules to the apache-badbots.conf file, and every time I have the doubt that it no longer works...
For example, this is my current apache-badbots.conf file:
[Definition]
badbotscustom = MQQBrowser|LieBaoFast|Mb2345Browser|zh-CN|python-requests|LinkpadBot|MegaIndex|Buck|SemrushBot|SeznamBot|JobboerseBot|AhrefsBot|AhrefsBot/6.1|MJ12bot|[email protected]|SemrushBot/6~bl|cortex|Cliqzbot|Baiduspider|serpstatbot|Go 1.1 package http|Python-urllib|StormCrawler|archive.org_bot|CCBot|BLEXBot|ltx71|DotBot|EmailCollector|WebEMailExtrac|Track$
badbots = Atomic_Email_Hunter/4\.0|atSpider/1\.0|autoemailspider|bwh3_user_agent|China Local Browse 2\.6|ContactBot/0\.2|ContentSmartz|DataCha0s/2\.0|DBrowse 1\.4b|DBrow$
#failregex = ^<HOST> -.*"(GET|POST|HEAD).*HTTP.*"(?:%(badbots)s|%(badbotscustom)s)"$
failregex = ^<HOST> -.*"(GET|POST).*HTTP.*".*(?:%(badbots)s|%(badbotscustom)s).*"$
ignoreregex =
datepattern = ^[^\[]*\[({DATE})
{^LN-BEG}
Yesterday I added "MQQBrowser | LieBaoFast | Mb2345Browser | zh-CN" and today I see a lot of MQQBrowser and LieBaoFast in my access logs.
sudo cat /var/log/apache2/access.log | awk -F\" '{print $6}' | sort | uniq -c | sort -n
...
3408 Mozilla/5.0(Linux;U;Android 5.1.1;zh-CN;OPPO A33 Build/LMY47V) AppleWebKit/537.36(KHTML,like Gecko) Version/4.0 Chrome/40.0.2214.89 UCBrowser/11.7.0.953 Mobile Safari/537.36
3418 Mozilla/5.0(Linux;Android 5.1.1;OPPO A33 Build/LMY47V;wv) AppleWebKit/537.36(KHTML,link Gecko) Version/4.0 Chrome/42.0.2311.138 Mobile Safari/537.36 Mb2345Browser/9.0
3444 Mozilla/5.0 (Linux; Android 7.0; FRD-AL00 Build/HUAWEIFRD-AL00; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/53.0.2785.49 Mobile MQQBrowser/6.2 TBS/043602 Safari/537.36 MicroMessenger/6.5.16.1120 NetType/WIFI Language/zh_CN
3473 Mozilla/5.0(Linux;Android 5.1.1;OPPO A33 Build/LMY47V;wv) AppleWebKit/537.36(KHTML,link Gecko) Version/4.0 Chrome/43.0.2357.121 Mobile Safari/537.36 LieBaoFast/4.51.3
What's wrong? It's working? Is there a way to tell if there's an error and what is the error?
I update because it still does not understand something, for example today I found other bots that should be banned in my logs.
Just to understand, this filter looks for the string I add in apache-badbots.conf in the server's access .log, and if it finds it adds a rule to fail2ban, right?
atSpider/1\.0" have all these slashes? China Local Browse 2\.6|DataCha0s/2\.0|DBrowse 1\.4b)Go 1.1 package http" are correct or generate an error?
For a while, when I access my ubuntu 18.04 server on vultr, by clicking the up arrow, I no longer see the bash history. It is always empty, even if I write some commands, then I exit the server and re-enter, the bash history is empty again.
I immediately thought it was the fault of some updates.
But the other day I noticed new hidden folders in my /www directory
/www
.local
.config
.ssh
and a new file: .bash_history Inside the file is this:
su
sudo cat ssh_host_rsa_key
ssh localhost
ssh localhost -p 65262
ssh localhost -p 65262 -u root
ssh root@localhost:65262
ssh [email protected]:65262
ssh root@localhost -p 65262
ssh root@localhost -p 65262 -i /etc/ssh/ssh_host_rsa_key
ssh root@localhost -p 65262 -i "/etc/ssh/ssh_host_rsa_key"
su root
su myuser
65262 is my old ssh port, I changed it, just for sure.
What's happening to my server? Is there anyone playing with it without my knowing it?
On my server I disabled the ssh login with password, disabled the root user, installed fail2ban with filter on ssh, changed the ssh port.
And so far I haven't seen any changes to my sites or my files.
I can't understand if it's just an update and a normal server behavior or if it's something more dangerous. Can you help me understand?
Update
I pasted the echo in a terminal, this is the result, but I have no idea if it is correct, wrong or if there is something strange.
HISTFILE=
HISTFILESIZE=2000
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
PS1=[\e]0;\u@\h: \w\a]${debian_chroot:+($debian_chroot)}\[\033[01;32m\]\u@\h\[\033[00m\]:\[\033[01;34m\]\w\[\033[00m\]\$
if I write this:
sudo nano ~/.bash_logout
I see this:
# ~/.bash_logout: executed by bash(1) when login shell exits.
# when leaving the console clear the screen to increase privacy
if [ "$SHLVL" = 1 ]; then
[ -x /usr/bin/clear_console ] && /usr/bin/clear_console -q
fi
It's normal???
And .profile comments say this, and seem to use the new dir I saw on my www:
# ~/.profile: executed by the command interpreter for login shells.
# This file is not read by bash(1), if ~/.bash_profile or ~/.bash_login
# exists.
# see /usr/share/doc/bash/examples/startup-files for examples.
# the files are located in the bash-doc package.
# the default umask is set in /etc/profile; for setting the umask
# for ssh logins, install and configure the libpam-umask package.
#umask 022
# if running bash
if [ -n "$BASH_VERSION" ]; then
# include .bashrc if it exists
if [ -f "$HOME/.bashrc" ]; then
. "$HOME/.bashrc"
fi
fi
# set PATH so it includes user's private bin if it exists
if [ -d "$HOME/bin" ] ; then
PATH="$HOME/bin:$PATH"
fi
# set PATH so it includes user's private bin if it exists
if [ -d "$HOME/.local/bin" ] ; then
PATH="$HOME/.local/bin:$PATH"
fi
At some point yesterday the functions: file_get_contents, get_headers, and curl have stopped working with error: Connection refused. Ex:
Curl: (7) Failed to connect to www.alebalweb.com port 443: Connection refused.
I do not understand why, I was just uploading files and no one touched anything on the server. Until a moment before everything worked and it worked for years. With Curl, I do the cronjob of my site once a day, and never had any problems. I thought the SSL certificates had expired, but it seems not. The site is online and it works, if I try to access the site via curl from my local PC, curl works smoothly. I thought I was banned at UFW or fail2ban for some reason, but I changed my local IP and nothing changed. And anyway I see a lot of errors in the Apache logs, from many different IPs. The problem occurs on all sites on my server.
I really can't understand what could have happened, everything worked perfectly until yesterday afternoon.
The Server is Ubuntu 18.04 on vultr.
I also found some strange errors in syslog, probably related.
Apr 8 06:25:03 Vultr_LA_18 rsyslogd: [origin software="rsyslogd" swVersion="8.32.0" x-pid="689" x-info="http://www.rsyslog.com"] rsyslogd was HUPed
Apr 8 06:25:07 Vultr_LA_18 popularity-contest: unable to submit report to https://popcon.ubuntu.com/popcon-submit.cgi.
Apr 8 06:25:07 Vultr_LA_18 popularity-contest: unable to submit report.
Apr 8 06:32:04 Vultr_LA_18 rsyslogd: unexpected GnuTLS error -53 - this could be caused by a broken connection. GnuTLS reports: Error in the push function. [v8.32.0$
Apr 8 06:32:04 Vultr_LA_18 rsyslogd: omfwd: TCPSendBuf error -2078, destruct TCP Connection to log.nixstats.com:10514 [v8.32.0 try http://www.rsyslog.com/e/2078 ]
Apr 8 06:32:04 Vultr_LA_18 rsyslogd: action 'action 1' suspended (module 'builtin:omfwd'), retry 0. There should be messages before this one giving the reason for sus$
Apr 8 06:32:04 Vultr_LA_18 rsyslogd: action 'action 1' resumed (module 'builtin:omfwd') [v8.32.0 try http://www.rsyslog.com/e/2359 ]
Apr 8 06:36:06 Vultr_LA_18 systemd-resolved[631]: Server returned error NXDOMAIN, mitigating potential DNS violation DVE-2018-0001, retrying transaction with reduced $
How can I create a filter to block these with fail2ban?
476 Mozilla/5.0 (compatible; BLEXBot/1.0; +http://webmeup-crawler.com/)
892 ltx71 - (http://ltx71.com/)
5367 Mozilla/5.0 (compatible; DotBot/1.1; http://www.opensiteexplorer.org/dotbot, [email protected])
6449 Barkrowler/0.9 (+http://www.exensa.com/crawl)
This list come out from this:
sudo cat /var/log/apache2/access.log | awk -F\" '{print $6}' | sort | uniq -c | sort -n
I've tried apache-badbot.conf, but it does not seem to work ...
For security reasons I would like to disable root access via ssh.
I created a new user (user1) with administrative permissions.
adduser user1
usermod -aG sudo user1
and assigned the /www directory to this user.
sudo chown -R $USER:$USER /var/www/
sudo chmod -R 755 /var/www
(My folders structure is www/site1.com, www/site2.com, etc.)
My sites need to write some files (such as sitemaps, rss feeds, etc.) so I set the permissions of the www directory to:
sudo chown -R www-data:www-data /var/www
sudo chmod -R 755 /var/www
Now, however, user user1 works perfectly via shell with the sudo command, but can no longer add/edit/delete files and folders in the /www directory and its subdirectories via sftp.
I read many guides, how to set up apache permissions to increase security, to share administration with other users, etc. etc. etc.
But I still did not understand how to solve my problem.
Currently to be able to handle files on my server via sftp I have to use the root user, with peace of mind for security.
Did I miss something about setting user or folders permissions?
In my /filter.d/ directory of Fail2ban there are many filters.
3proxy.conf asterisk.conf ejabberd-auth.conf lighttpd-auth.conf oracleims.conf selinux-common.conf tine20.conf
apache-auth.conf badbot.conf exim-common.conf mongodb-auth.conf pam-generic.conf selinux-ssh.conf uwimap-auth.conf
apache-badbots.conf botsearch-common.conf exim.conf monit.conf perdition.conf sendmail-auth.conf vsftpd.conf
apache-botsearch.conf common.conf exim-spam.conf murmur.conf phpmyadmin-syslog.conf sendmail-reject.conf webmin-auth.conf
apache-common.conf counter-strike.conf freeswitch.conf mysqld-auth.conf php-url-fopen.conf sieve.conf wp-admin.conf
apache-fakegooglebot.conf courier-auth.conf froxlor-auth.conf nagios.conf portsentry.conf slapd.conf wp-login.conf
apache-modsecurity.conf courier-smtp.conf groupoffice.conf named-refused.conf postfix.conf sogo-auth.conf wuftpd.conf
apache-nohome.conf cyrus-imap.conf gssftpd.conf nginx-botsearch.conf proftpd.conf solid-pop3d.conf xinetd-fail.conf
apache-noscript.conf directadmin.conf guacamole.conf nginx-http-auth.conf pure-ftpd.conf squid.conf zoneminder.conf
apache-overflows.conf domino-smtp.conf haproxy-http-auth.conf nginx-limit-req.conf qmail.conf squirrelmail.conf
apache-pass.conf dovecot.conf horde.conf nsd.conf recidive.conf sshd.conf
apache-shellshock.conf dropbear.conf ignorecommands openhab.conf roundcube-auth.conf stunnel.conf
assp.conf drupal-auth.conf kerio.conf openwebmail.conf screensharingd.conf suhosin.conf
Some I can understand more or less what they do from the name, others I have no idea.
Is there an online page with 2 lines of description for each filter?
I searched online and on the site of Fail2ban, but even with the name of the filter finds little.
In the access logs of my server I find thousands of attempts to access wp-login.php, but none of the sites on my server is wordpress...
37.19.108.121 - - [19/Dec/2018:02:31:10 +0000] "GET /wp-login.php HTTP/1.1" 301 525 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.1"
37.19.108.121 - - [19/Dec/2018:02:31:11 +0000] "GET /wp-login.php HTTP/1.1" 403 14611 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.1"
179.189.156.125 - - [19/Dec/2018:02:36:14 +0000] "GET /wp-login.php HTTP/1.1" 301 527 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.1"
179.189.156.125 - - [19/Dec/2018:02:36:15 +0000] "GET /wp-login.php HTTP/1.1" 403 3732 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.1"
58.26.43.39 - - [19/Dec/2018:02:40:04 +0000] "GET /wp-login.php HTTP/1.1" 301 535 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.1"
58.26.43.39 - - [19/Dec/2018:02:40:05 +0000] "GET /wp-login.php HTTP/1.1" 403 3748 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.1"
37.19.108.121 - - [19/Dec/2018:02:40:34 +0000] "GET /wp-login.php HTTP/1.1" 301 535 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.1"
37.19.108.121 - - [19/Dec/2018:02:40:36 +0000] "GET /wp-login.php HTTP/1.1" 403 14663 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.1"
116.199.102.237 - - [19/Dec/2018:02:48:04 +0000] "GET /wp-login.php HTTP/1.1" 301 525 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.1"
116.199.102.237 - - [19/Dec/2018:02:48:04 +0000] "GET /wp-login.php HTTP/1.1" 403 3907 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.1"
201.92.187.24 - - [19/Dec/2018:02:48:32 +0000] "GET /wp-login.php HTTP/1.1" 301 537 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.1"
201.92.187.24 - - [19/Dec/2018:02:48:33 +0000] "GET /wp-login.php HTTP/1.1" 403 16139 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.1"
58.26.43.39 - - [19/Dec/2018:02:52:44 +0000] "GET /wp-login.php HTTP/1.1" 301 533 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.1"
58.26.43.39 - - [19/Dec/2018:02:52:45 +0000] "GET /wp-login.php HTTP/1.1" 403 3565 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.1"
179.215.42.142 - - [19/Dec/2018:03:01:40 +0000] "GET /wp-login.php HTTP/1.1" 301 521 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.1"
179.215.42.142 - - [19/Dec/2018:03:01:41 +0000] "GET /wp-login.php HTTP/1.1" 403 21504 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.1"
In my htaccess I added this rule:
RewriteCond %{REQUEST_URI} wp-login.php$ [NC]
RewriteRule . - [F,L]
But my server is always overloaded anyway.
I started to add the IPs to UFW, but they are now thousands and every day I find new ones. And probably the IPs will be dynamic and never end...
What is the best way to block these attacks without overloading the server?
I tried Fail2ban, after some online search I created this filter:
/etc/fail2ban/filter.d/wp-login.conf
[Definition]
failregex = ^<HOST> .* "POST .*wp-login.php
^<HOST> .* "GET .*wp-login.php
^<HOST> .* "POST .*xmlrpc.php
ignoreregex =
And added this to jail.local
[wp-login]
enabled = true
port = http,https
filter = wp-login
action = iptables-multiport[name=wp-login, port="http,https", protocol=tcp]
logpath = /var/log/apache2/access.log
maxretry = 1
findtime = 600
But at some point was banned my IP... but I never went on wp-login.php, What's wrong? Can someone help me understand and configure fail2ban?
I removed the 3 lines and I left only one
failregex = ^<HOST> .* "GET .*/wp-login.php
seems to work better... I have no idea why, the guides I read say that you can put more rules on different lines...???
I added this in my htaccess:
RewriteEngine On
RewriteBase /
RewriteCond %{HTTP_USER_AGENT} ^SemrushBot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Yandex [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^rogerbot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^exabot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^MJ12bot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^dotbot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^gigabot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^AhrefsBot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^CCBot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^spbot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Mjbot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^MauiBot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^SeznamBot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Barkrowler [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^MegaIndex [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^AlphaBot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Cliqzbot [NC,OR]
RewriteCond %{REQUEST_URI} wp-login.php$ [NC]
#RewriteRule .* - [L,F]
RewriteRule . - [F,L]
#RewriteRule ^.* - [F,L]
Looking for example MJ12bot in my access logs with this:
grep --text MJ12bot /var/log/apache2/access.log
I should not see towards the end HTTP/1.1 403?
Instead I see:
HTTP/1.1" 200 10841 "-" "Mozilla/5.0 (compatible; MJ12bot/v1.4.8; http://mj12bot.com/)"
is something not working? What?
for wp-login.php seems to work, I see:
HTTP/1.1" 403 24328 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.1"
What's wrong?
I occasionally use this line in my terminal to see which user-agent is using my server more.
cat /var/log/apache2/access.log | awk -F\" '{print $6}' | sort | uniq -c | sort -n
Scans the access.log and show in ascending order the user-agents that it has found several times.
The result is something like this:
10283 Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.1
23247 Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.93 Safari/537.36
40063 MauiBot ([email protected])
143724 Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
192741 Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)
I have no idea how it works, so long ago I found it somewhere or someone gave me the date, I do not remember.
Anyway, is possible do the same thing but ordering for IP?
My access.log are full of:
Mozilla/5.0 (compatible; DotBot/1.1; http://www.opensiteexplorer.org/dotbot, [email protected])
CCBot/2.0 (http://commoncrawl.org/faq/)
Mozilla/5.0 (compatible; spbot/5.0.3; +http://OpenLinkProfiler.org/bot )
Mozilla/5.0 (compatible; MJ12bot/v1.4.8; http://mj12bot.com/)
Mozilla/5.0 (compatible; SeznamBot/3.2; +http://napoveda.seznam.cz/en/seznambot-intro/)
and of:
85.55.237.132 - - [16/Jul/2018:06:30:44 +0000] "GET /wp-login.php HTTP/1.1" 404 11462 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.1"
51.253.65.140 - - [16/Jul/2018:06:30:47 +0000] "GET /wp-login.php HTTP/1.1" 404 7645 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.1"
113.161.75.37 - - [16/Jul/2018:06:30:57 +0000] "GET /wp-login.php HTTP/1.1" 404 12145 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.1"
114.94.208.2 - - [16/Jul/2018:06:31:04 +0000] "GET /wp-login.php HTTP/1.1" 404 11096 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.1"
112.134.172.163 - - [16/Jul/2018:06:31:07 +0000] "GET /wp-login.php HTTP/1.1" 404 11177 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.1"
144.48.75.34 - - [16/Jul/2018:06:31:13 +0000] "GET /wp-login.php HTTP/1.1" 404 7736 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.1"
And none of my sites is wordpress.
Currently I block access to these via UFW blocking the IPs, but they have become thousands and every day the number increases.
Is there any way to say to Ubuntu for example:
If the user-agent (or requested URL) contains "Mjbot" block it.
I've seen several tools like Fail2ban, modsecurity, etc, but they seem to complicate things rather than simplify them.
I'm looking for a really simple solution.
I updated my server from 16.04 to 18.04 a few days ago, today I found this:
19 packages can be updated.
18 updates are security updates.
I did:
sudo apt-get update
sudo apt-get upgrade
Reading package lists... Done
Building dependency tree
Reading state information... Done
Calculating upgrade... Done
The following packages have been kept back:
libapache2-mod-php7.2 php7.2-bcmath php7.2-bz2 php7.2-cgi php7.2-cli php7.2-common php7.2-fpm php7.2-gd php7.2-intl php7.2-json php7.2-mbstring php7.2-mysql php7.2-opcache php7.2-readline php7.2-xml php7.2-xmlrpc php7.2-zip
The following packages will be upgraded:
php7.2 snap-confine snapd ubuntu-core-launcher
4 upgraded, 0 newly installed, 0 to remove and 17 not upgraded.
Need to get 12.2 MB of archives.
After this operation, 21.5 kB of additional disk space will be used.
Do you want to continue? [Y/n] y
Get:1 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 ubuntu-core-launcher amd64 2.32.8+18.04 [1,644 B]
Get:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 snapd amd64 2.32.8+18.04 [12.2 MB]
Get:3 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 snap-confine amd64 2.32.8+18.04 [1,792 B]
Get:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 php7.2 all 7.2.5-0ubuntu0.18.04.1 [9,240 B]
Fetched 12.2 MB in 2s (6,527 kB/s)
(Reading database ... 81195 files and directories currently installed.)
Preparing to unpack .../ubuntu-core-launcher_2.32.8+18.04_amd64.deb ...
Unpacking ubuntu-core-launcher (2.32.8+18.04) over (2.32.5+18.04) ...
Preparing to unpack .../snapd_2.32.8+18.04_amd64.deb ...
Unpacking snapd (2.32.8+18.04) over (2.32.5+18.04) ...
Preparing to unpack .../snap-confine_2.32.8+18.04_amd64.deb ...
Unpacking snap-confine (2.32.8+18.04) over (2.32.5+18.04) ...
Preparing to unpack .../php7.2_7.2.5-0ubuntu0.18.04.1_all.deb ...
Unpacking php7.2 (7.2.5-0ubuntu0.18.04.1) over (7.2.4-1+ubuntu16.04.1+deb.sury.org+1) ...
Setting up snapd (2.32.8+18.04) ...
Created symlink /etc/systemd/system/multi-user.target.wants/snapd.seeded.service → /lib/systemd/system/snapd.seeded.service.
Created symlink /etc/systemd/system/cloud-final.service.wants/snapd.seeded.service → /lib/systemd/system/snapd.seeded.service.
snapd.snap-repair.service is a disabled or a static unit, not starting it.
Setting up php7.2 (7.2.5-0ubuntu0.18.04.1) ...
Processing triggers for man-db (2.8.3-2) ...
Setting up ubuntu-core-launcher (2.32.8+18.04) ...
Setting up snap-confine (2.32.8+18.04) ...
Now:
18 packages can be updated.
17 updates are security updates.
Reading package lists... Done
Building dependency tree
Reading state information... Done
Calculating upgrade... Done
The following packages have been kept back:
libapache2-mod-php7.2 php7.2-bcmath php7.2-bz2 php7.2-cgi php7.2-cli php7.2-common php7.2-fpm php7.2-gd php7.2-intl php7.2-json php7.2-mbstring php7.2-mysql php7.2-opcache php7.2-readline php7.2-xml php7.2-xmlrpc php7.2-zip
0 upgraded, 0 newly installed, 0 to remove and 17 not upgraded.
How is that possible? What did I miss? Is a web server updated a few days ago, the update is about PHP which is very important for a web server. What can I do? What should I do? Will they update later along with other updates? Is this behavior normal after a few days from the version update? My server is in Vultr, maybe something is missing in Vultr?
I often search in the access.log for IP of services that do not interest me (E. Semrush, Mj12bot, Megaindex etc.) and add them to UFW.
But often adding the rule to UFW responds like this:
ufw deny from 46.229.173.137 to any;
Skipping adding existing Rule
The rule existed but was ignored?
After adding a rule I always do also:
ufw reload
And sometimes I also restart the server.
My server is on Vultr, Ubuntu 16.04 x64
Why are UFW's rules ignored? Is it Vultr's fault? Ubuntu's fault? UFW's fault?
ufw status here: https://pastebin.com/6HS1ZgiA (Too long to post here)
Example IP in acces.log: http://i63.tinypic.com/dnetu1.png
