ticktockhouse's questions

Here is my logs directory:

root@my-server:/var/www/vhosts/my-website/logs# ll
total 1468016
-rw-r--r-- 1 my-website webuser        95719295 Mar 19 18:04 my-website-access.log
-rw-r--r-- 1 my-website webuser        21130452 Apr 27  2019 my-website-access.log.4.gz
-rw-r--r-- 1 my-website webuser          831892 Mar 17 15:39 my-website-error.log
-rw-r--r-- 1 root       root         1238469639 Mar 19 18:04 my-website-ssl-access.log
-rw-r--r-- 1 root       root                 20 Mar 19 17:55 my-website-ssl-access.log.3.gz
-rw-r--r-- 1 root       root           16976113 Mar  2  2019 my-website-ssl-access.log.9.gz
-rw-r--r-- 1 root       root          102746845 Mar 19 18:04 my-website-ssl-error.log
-rw-r--r-- 1 my-website webuser        21089307 Mar 19 18:03 test.my-website-access.log
-rw-r--r-- 1 my-website webuser         6243491 Mar 17 18:48 test.my-website-error.log

Note the massive > 1GB file sitting there. Here is my logrotate config file:

/var/www/vhosts/*/logs/*.log {
  rotate 8
  daily
  size 256M
  compress
  delaycompress
  sharedscripts

  postrotate
    /bin/systemctl reload apache2  > /dev/null
  endscript
}

When I run the command:

logrotate -vv -f /etc/logrotate.d/vhost

it appears to be working, I get these messages (amongst a lot of other noise):

considering log /var/www/vhosts/my-website/logs/my-website-ssl-access.log
  log needs rotating
...
rotating log /var/www/vhosts/my-website/logs/my-website-ssl-access.log, log->rotateCount is 8
dateext suffix '-20200319'
glob pattern '-[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]'
previous log /var/www/vhosts/my-website/logs/my-website-ssl-access.log.1 does not exist
renaming /var/www/vhosts/my-website/logs/my-website-ssl-access.log.8.gz to /var/www/vhosts/my-website/logs/my-website-ssl-access.log.9.gz (rotatecount 8, logstart 1, i 8), 
old log /var/www/vhosts/my-website/logs/my-website-ssl-access.log.8.gz does not exist
renaming /var/www/vhosts/my-website/logs/my-website-ssl-access.log.7.gz to /var/www/vhosts/my-website/logs/my-website-ssl-access.log.8.gz (rotatecount 8, logstart 1, i 7), 
old log /var/www/vhosts/my-website/logs/my-website-ssl-access.log.7.gz does not exist
renaming /var/www/vhosts/my-website/logs/my-website-ssl-access.log.6.gz to /var/www/vhosts/my-website/logs/my-website-ssl-access.log.7.gz (rotatecount 8, logstart 1, i 6), 
old log /var/www/vhosts/my-website/logs/my-website-ssl-access.log.6.gz does not exist
renaming /var/www/vhosts/my-website/logs/my-website-ssl-access.log.5.gz to /var/www/vhosts/my-website/logs/my-website-ssl-access.log.6.gz (rotatecount 8, logstart 1, i 5), 
old log /var/www/vhosts/my-website/logs/my-website-ssl-access.log.5.gz does not exist
renaming /var/www/vhosts/my-website/logs/my-website-ssl-access.log.4.gz to /var/www/vhosts/my-website/logs/my-website-ssl-access.log.5.gz (rotatecount 8, logstart 1, i 4), 
old log /var/www/vhosts/my-website/logs/my-website-ssl-access.log.4.gz does not exist
renaming /var/www/vhosts/my-website/logs/my-website-ssl-access.log.3.gz to /var/www/vhosts/my-website/logs/my-website-ssl-access.log.4.gz (rotatecount 8, logstart 1, i 3), 
old log /var/www/vhosts/my-website/logs/my-website-ssl-access.log.3.gz does not exist
renaming /var/www/vhosts/my-website/logs/my-website-ssl-access.log.2.gz to /var/www/vhosts/my-website/logs/my-website-ssl-access.log.3.gz (rotatecount 8, logstart 1, i 2), 
renaming /var/www/vhosts/my-website/logs/my-website-ssl-access.log.1.gz to /var/www/vhosts/my-website/logs/my-website-ssl-access.log.2.gz (rotatecount 8, logstart 1, i 1), 
old log /var/www/vhosts/my-website/logs/my-website-ssl-access.log.1.gz does not exist
renaming /var/www/vhosts/my-website/logs/my-website-ssl-access.log.0.gz to /var/www/vhosts/my-website/logs/my-website-ssl-access.log.1.gz (rotatecount 8, logstart 1, i 0), 
old log /var/www/vhosts/my-website/logs/my-website-ssl-access.log.0.gz does not exist
log /var/www/vhosts/my-website/logs/my-website-ssl-access.log.9.gz doesn't exist -- won't try to dispose of it
rotating log /var/www/vhosts/my-website/logs/my-website-ssl-error.log, log->rotateCount is 8
dateext suffix '-20200319'

...but the large log file does not appear to be being rotated despite logrotate apparently saying it has:

-rw-r--r-- 1 root       root         1238469639 Mar 19 18:04 my-website-ssl-access.log

Any idea what this would be happening?

My task is to create a postfix config that accepts all mail from all domains, recipients, etc. and forwards them to a catchall mailbox. This may be further filtered in future and indeed the config will need to be latered in another environment so that mail goes through to the intended recipient.

Here's an abbreviated main.cf:

....
myhostname = mailserver.fqdn
alias_maps = hash:/etc/aliases
alias_database = hash:/etc/aliases
mydestination =
myorigin = localhost
relayhost =
mynetworks = 127.0.0.0/8 [::ffff:127.0.0.0]/104 [::1]/128 192.168.5.0/24
mailbox_size_limit = 0
recipient_delimiter = +
inet_interfaces = 192.168.5.43, 127.0.0.1
inet_protocols = all

virtual_alias_maps = pcre:/etc/postfix/virtual

virtual_mailbox_base = /var/vmail
virtual_minimum_uid = 2222
virtual_transport = virtual
virtual_uid_maps = static:2222
virtual_gid_maps = static:2222
....

/etc/postfix/virtual:

/.*/     catchall
/catchall\@localhost/ catchall

Other machines are configured to use this mail server as a relayhost, and when mail makes it to here, this appears in the log:

Nov 28 10:08:10 mailserver.fqdn postfix/error[23768]: 5EA9BBD9F5: to=<catchall@localhost>, orig_to=<[email protected]>, relay=none, delay=0, delays=0/0/0/0, dsn=5.1.1, status=bounced (User unknown in virtual alias table)

I'm probably missing something obvious, is it the presence of alias_maps = hash:/etc/aliases?

Also, if all goes well, can I expect the catchall mailbox to be created automatically, or is there another config item needed?

I would like my postfix server to act as a relay/smarthost for some clients, but would like to deliver all mail to a local (Maildir-style) mailbox instead of sending on to the intended recipient. Sounds a bit weird, I know, but it is intended for testing purposes on our test environments.

For our live environment, I would like to deliver mail normally.

My current main.cf looks like this:

myhostname = mymail.mydomain.com
alias_maps = hash:/etc/aliases
alias_database = hash:/etc/aliases
mydestination = pcre:/etc/postfix/mydestinations
relayhost = 
mynetworks = 127.0.0.0/8 [::ffff:127.0.0.0]/104 [::1]/128 192.168.5.0/24
mailbox_size_limit = 0
recipient_delimiter = +
inet_interfaces = 192.168.5.43, 127.0.0.1
inet_protocols = all
virtual_alias_maps = hash:/etc/postfix/virtual

default_process_limit = 100
smtpd_client_connection_count_limit = 10
smtpd_client_connection_rate_limit = 30
queue_minfree = 20971520
header_size_limit = 51200
message_size_limit = 10485760
smtpd_recipient_limit = 100

smtpd_tls_cert_file=/etc/ssl/certs/mailcert.crt 
smtpd_tls_key_file=/etc/ssl/private/mailcert.key
smtpd_use_tls=yes
smtpd_tls_session_cache_database = btree:${data_directory}/smtpd_scache
smtp_tls_session_cache_database = btree:${data_directory}/smtp_scache
smtpd_tls_security_level=may
smtpd_tls_protocols = !SSLv2, !SSLv3

/etc/postfix/mydestinations:

/^mymail.mydomain.com$/          ACCEPT
/^localhost.localdomain$/           ACCEPT
/^localhost$/                       ACCEPT
/^.*\.mydomain\.com$/       ACCEPT

/etc/postfix/virtual:

@mydomain.com catchall

I've been told this is possible, but can't really find any specific info as to how to achieve it..

I am running rsnapshot, and certain backups are taking up a lot of space. I've run rsnapshot-diff, and it gives results like this:

Between /home/backups/hourly.4/mybackupdir/ and /home/backups/daily.0/mybackupdir/:
  1705 were added, taking 15518770836 bytes;
  1705 were removed, saving 15518443844 bytes;

...but an rsnapshot du shows that the backup set takes up the advertised 15 GB or so. If I tell it to produce verbose rsnaphot-diff output, it appears to show that the same files were added and removed, so I'm somewhat confused as to why it's using up so much space - is there something wrong with the rsnapshot process that's causing it to use the extra space or something?

One of the servers I look after appears to be participating in brute force attacks against Wordpress installations.

I've been on the receiving end of this many times, so am very familiar with steps that can be taken to prevent this. What I'm struggling with, however, is detecting outgoing attacks. The server is a typical Apache server with a number of vhosts on it - this is where the complication comes of course - if there was just one on there, it wouldn't be as difficult!

I'm currently using tcpflow to log traffic going from any port on this server to port 80 on any other machine using this command:

tcpflow -i eth0 dst port 80 and src host <my_servers_ip> and port not 22

I've found this preferable to tcpdump. Looking through its output can be somewhat brain-melting after a while :) tcpflow puts each request into a separate file..

Here is some output from a file which I believe to be suspicious activity:

POST /wp-login.php HTTP/1.1
User-Agent: Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Host: somedomain.com
Accept: */*
Cookie: wordpress_test_cookie=WP+Cookie+check
Content-Length: 97
Content-Type: application/x-www-form-urlencoded

log=jacklyn&pwd=london&wp-submit=Log+In&redirect_to=http://somedomain.com/wp-admin/tes1a0&testcookie=1

Please note, I've obfuscated the "Host:" above, I believe that's the host being attacked (is this correct?).

So my question really, is how do I go about detecting the vhost that is generating this malicious traffic? If I can do that, I can let my client know, and he can take steps to investigate the site and make the necessary changes to stop it..

Any solutions very gratefully received :)

I've got some troublesome hosts trying it on with an SSH server I run, and I'm trying to ban them using fail2ban. Problem is, I haven't done much work with regexes, and even less with Python regexes.

Here are the troublesome lines in my auth.log:

Nov 19 18:58:17 myhost sshd[48272]: Connection from xxx.xxx.xxx.xxx port 3284 on my.host.ip.address port 22
Nov 19 18:58:21 myhost sshd[48272]: fatal: Read from socket failed: Connection reset by peer [preauth]

I want to grab both lines in the regex, I've seen in other posts about how to do multiline stuff, but at the moment I can't even get it to match the first line! Here's a snippet from my *.conf file:

[INCLUDES]

# Read common prefixes. If any customizations available -- read them from
# common.local
before = common.conf

[Init]

maxlines = 2

[Definition]

_daemon = sshd

failregex = ^%(__prefix_line)s^Connection from <HOST>*$

I understand that the "__prefix_line" is designed to catch the first "myhost sshd[PID]" bit, but all I when I run "fail2ban-regex" is:

Results
=======

Failregex: 0 total

Ignoreregex: 0 total

Date template hits:
|- [# of hits] date format
|  [115124] MONTH Day Hour:Minute:Second
`-

Lines: 115124 lines, 0 ignored, 0 matched, 115124 missed

Does anyone have any ideas?

Thanks in advance!

I'm in the somewhat embarrassing position of having unintentionally deleted multiple TB of important data via Puppet, and I'm just trying to understand why this might have happened.

Firstly, I'm pretty sure the reason it's gone (as in unrecoverable except via backups) is:

File { backup => false }

in my site.pp. The nodes were set up to hard mount something via NFS, so a mount point /mount, and a line in fstab like this:

nfsserver:/mount   /mount  nfs   <options> 0 0

I wanted to get rid of the mount, and replace it with a symlink to the same eventual location (though a different path).

My puppet manifest looked like this:

class symlinks::linkdirtest ( ) { 
        file { '/mount':
            ensure => "link",
            target => "/anotherdir/mount",
        }
        mount { "/mount":
            ensure => "absent",
        }
}

This yielded the following when doing a puppet run:

notice: /Stage[main]/Symlinks::Linkdirtest/File[/mount]: Not removing directory; use 'force' to override

So, I duly (or stupidly) added:

class symlinks::linkdirtest ( ) { 
        file { '/mount':
            ensure => "link",
            target => "/anotherdir/mount",
                    force => "true",
        }
....

And lo and behold, puppet proceeded to consign the contents of the all-important mount to oblivion, while the mount point itself remained.

Any idea why this might have happened?

Thanks

Please bear with me, I've searched as best I can with no apparent solutions.

I'm behind my companies firewall, we have a couple of proxy servers that we can use to get out onto the big wide internet. My company's IT group has kindly mirrored the main Debian repos inside the WAN.

I would like to use a couple of repos outside the WAN (i.e. the Internet). Is it possible to tell apt-get to use a proxy for these, and not for the others?

With both enabled and a proxy defined in apt.conf, it barfs on the stuff inside the firewall, and without a proxy defined it barfs on the sources outside the firewall.

There is some mention in the man page for apt.conf, but putting a proxy statement in the source.list file doesn't work (unless I've got the syntax wrong).

Thanks

Jerry

I have this problem with NRPE, all the stuff I've found so far on the net seems to point me at things I've already tried.

# /usr/local/nagios/plugins/check_nrpe -H nrpeclient

gives

NRPE v2.12

as expected.

Running the command by hand (as defined in nrpe.cfg on "nrpeclient", gives the expected response

nrpe.cfg:

command[check_openmanage]=/usr/lib/nagios/plugins/additional/check_openmanage -s -e   -b ctrl_driver=0 bat_charge

"Expected response"

But if I try to run the command from the Nagios server I get the following:

# /usr/local/nagios/plugins/check_nrpe -H comxps -c check_openmanage
NRPE: Unable to read output

Can anyone think of anywhere else I might have made a mistake with this? I've done the same thing on multiple other servers with no problem. The only difference I can think of with this is that this box is RHEL 5 based, whereas the others are RHEL 4 based.

Those two bits above that I've tested are the what most people seem to suggest when people have had this problem.

I should mention that I get a weird error in the logs when I restart nrpe:

nrpe[14534]: Unable to open config file '/usr/local/nagios/etc/nrpe.cfg' for reading 
nrpe[14534]: Continuing with errors...
nrpe[14535]: Starting up daemon
nrpe[14535]: Warning: Daemon is configured to accept command arguments from clients!
nrpe[14535]: Listening for connections on port 5666 
nrpe[14535]: Allowing connections from: bodbck,combck,nam-bck

Even though, it's plainly reading that /usr/local/nagios/etc/nrpe.cfg file to get the stuff it's talking about further down..

I must admit to being a complete syslog newbie. I haven't touched it until now, but I use its output on a daily basis, like most of us :)

I currently have a problem with the gmetad daemon spamming my /var/log/messages file with messages like the following:

Sep 1 10:09:49 monitor /usr/sbin/gmetad[31752]: RRD_update (/var/lib/ganglia/rrds/Machines/SummaryInfo/example.rrd): illegal attempt to u pdate using time 1314868188 when last update time is 1314868188 (minimum one second step).

I'm not too worried about the error, what I want to do is redirect these particular messages to another log file (e.g /var/log/gmetad.log). The problem is that there doesn't seem to be any way to direct gmetad's logs to a certain place in its config (and documentation for this particular piece of software seems quite sparse). So I was wondering if it was possible to use the "local" facility in syslog to grab these messages and stick them somewhere else to leave /var/log/messages a bit cleaner.

Can anyone help? Does anyone have any examples for the "local" facilities?

The following command on Nagios's default "check_disk" command gives the following output:

# ./check_disk  -w 20% -c 10%  -p / --units=MB 
DISK OK - free space: / 17887 MB (47% inode=93%);| /=19481MB;31495;35432;0;39369

All well and good, but does anyone know what the last four numbers (separated by semicolons) mean? Is there any way, other than wrapping the command in a script, to tailor the output to something which makes sense at first glance?

Thanks in advance...