Giacomo1968's questions

So I am rebuilding a server from Ubuntu 12.04 to Ubuntu 14.04; not doing the upgrade path because I would rather build from scratch in this case. I have installed Munin as I have done before in Ubuntu 12.04 like this:

sudo aptitude install munin munin-node

Then I enabled the Apache modules for Munin like this:

sudo ln -s /usr/share/munin/plugins/apache_accesses /etc/munin/plugins/apache_accesses
sudo ln -s /usr/share/munin/plugins/apache_processes /etc/munin/plugins/apache_processes
sudo ln -s /usr/share/munin/plugins/apache_volume /etc/munin/plugins/apache_volume

Then I restarted the Munin node like this:

sudo service munin-node restart

And waited the requisite 5-10 minutes to get data and related graphs generated. And happily the graphs showed up! But unhappily, all the values were -nan, meaning somehow the data was not being processed by Munin; see screenshot below:

Usually this means Munin is having problems reaching the localhost’s Apache service-status page, but checking that URL on the system with Lynx shows everything works fine:

lynx http://localhost/server-status

So I am stumped as to what might have happened. Did a full system restart and 100% no improvement. What could be happening?

I am running this command to get a tally of failed login attempts from a server’s auth.log and it works well:

sudo cat /var/log/auth.{log,log.1} | grep 'Failed password' | grep sshd | awk '{print $1,$2}' | sort -k 1,1M -k 2n | uniq -c

But the issue is—like every server in the world—I have some GZipped archives created by log rotation that won’t be parsed by this command:

-rw-r----- 1 syslog adm  7822722 Oct 31 13:44 /var/log/auth.log
-rw-r----- 1 syslog adm 12532511 Oct 25 06:59 /var/log/auth.log.1
-rw-r----- 1 syslog adm  2250939 Oct 18 06:55 /var/log/auth.log.2.gz
-rw-r----- 1 syslog adm  2139669 Oct 11 07:06 /var/log/auth.log.3.gz
-rw-r----- 1 syslog adm  2769919 Oct  4 06:54 /var/log/auth.log.4.gz

To deal with those I have a variant of the command that uses zcat instead of cat but it’s a bit of a nuisance to have to run two commands to get this data. Is there any way to combine the cat and zcat commands into one command so I can get combined results? I know I could write some Bash script to filter through compressed versus uncompressed files, but I use this current one-liner on various servers and I want to have a one-liner I can quickly refer to and use when needed.

More details.

I’ve been doing some security auditing and firewall tweaking and have this command I run that gives me a nice tally of “Failed password” attempts from the auth.log files on an Ubuntu Linux machine:

sudo cat /var/log/auth.{log,log.1} | grep 'Failed password' | grep sshd | awk '{print $1,$2}' | sort -k 1,1M -k 2n | uniq -c

It works great! The output looks something like this:

 5909 Oct 18
13444 Oct 19
  351 Oct 20
  162 Oct 21
  499 Oct 22
  377 Oct 23
  145 Oct 24
10897 Oct 25
   76 Oct 26
   54 Oct 27
  310 Oct 28
 1024 Oct 29
  208 Oct 30
   30 Oct 31

And while this works well for uncompressed logs, since the logs rotate and get compressed there are always some GZip compressed files that would be nice to tally up as well. So I run this variant of the above command that uses zcat:

sudo zcat -q /var/log/auth.log* | grep 'Failed password' | grep sshd | awk '{print $1,$2}' | sort -k 1,1M -k 2n | uniq -c

And the output is something like this:

gzip: /var/log/auth.log: not in gzip format

gzip: /var/log/auth.log.1: not in gzip format
  10413 Sep 27
  15977 Sep 28
  12297 Sep 29
  14438 Sep 30
  23394 Oct 1
  12912 Oct 2
  15844 Oct 3
  19697 Oct 4
  15350 Oct 5
  12358 Oct 6
  12692 Oct 7
   8377 Oct 8
  10875 Oct 9
    565 Oct 10
  16027 Oct 11
  10422 Oct 12
   6808 Oct 13
  26891 Oct 14
   9493 Oct 15
   5138 Oct 16
   9415 Oct 17
   2226 Oct 18

As you can see, the output works when it works, but it would be nice if the cat and zcat variants of this command could simply be combined into one command. How can that be done?

Bonus Points:

These are not critical issues, but it would be nice if somehow they could be addressed in a solution:

• Note that at the top of the zcat output there are two errors from zcat attempting to process two uncompressed files. Suppressing that would be nice.
• Also note how the cat and zcat commands show data for October 18th; a day when log rotation happened. Would there be any way to add up those two values in the command? If not, I’m fine with having two lines of October 18th data with different values I would have to add up manually.

I recently installed Fail2Ban on an Ubuntu 12.04.5 server and it’s working great. Now I want to setup Munin so it can monitor it and generate graphs/reports. I found that Munin has a built-in Fail2Ban plugin that can be enabled like this:

sudo ln -s /usr/share/munin/plugins/fail2ban /etc/munin/plugins/fail2ban

Did that and restarted Munin like this:

sudo service munin-node restart

And waited 5-10 minutes for reports to be generated. And while there is a Fail2Ban area in the server report, all images are broken. Why? What can I do to fix this?

Basic question.

How can I install an upgraded version of Fail2Ban on my Ubuntu 12.04.5 LTS (Precise Pangolin) so I can have a recidive filter set. Ubuntu 12.04.5 LTS installs Fail2Ban 0.8.6 and I need Fail2Ban 0.8.7 to use the recidive filter.

More details.

I manage a bunch of Ubuntu 12.04.5 LTS (Precise Pangolin) servers and have installed Fail2Ban specifically to block repeated SSH brute force attempts. These attacks are targeted at root and the root user is disabled on all of these systems, but I just want to add another layer of protection to my mix as well as cut down on the added “noise” of having to sift through failed login attempts in my SSH auth.log.

Anyway, the version of Fail2Ban—version 0.8.6—that is installed for Ubuntu 12.04.5 LTS from the default repositories works great for the most part. But it seems to be missing support for blocking repeat offenders. And these servers are definitely getting a lot of persistent brute force attempts. So I want to make sure that these clowns are blocked for a longer time.

I checked this blog post titled, “Permanently Ban Repeat Offenders With fail2ban (UPDATED)” and it recommends using the recidive filter. The rest of that post shows how to set up a customized equivalent of recidive but I don’t have any desire to use a custom solution if a built in solution such as recidive exists.

But when I attempt to set the recidive filter in Fail2Ban 0.8.6 and restart the service, it fails saying the filter recidive doesn’t exist.

When I check the Fail2Ban changelog on GitHub I can see that recidive support was added around version 0.8.7:

Tom Hendrikx
 * [f94a121..] 'recidive' filter/jail to monitor fail2ban.conf to ban
   repeated offenders. Close gh-19

Well, isn’t that special. I’m using 0.8.6 and version 0.8.7 has support for this.

So how can I easily install anything newer than Fail2Ban 0.8.6 on Ubuntu 12.04.5 LTS? I tried to install version 0.9.3 by cloning the GitHub repository, switching to the Debian branch (since Ubuntu is Debian-based) and it seemed to work. But the startup service didn’t work and it also seems like that 0.9.3 won’t work with the version of IPTables I have installed in Ubuntu 12.04.5 LTS unless I tweak some config files or possibly upgrade IPTables itself.

Maybe there’s a way to tweak it all to work for Ubuntu 12.04.5 LTS but I don’t have the time or will to deal with that since all I want is recidive support and don’t care about anything else. What can I do?

The subject says it all. A client I have has a relatively ancient Xserve (2009 model I believe) that is running Mac OS X 10.6.8 Server. We’ve been able to remote into the system without issue in the past, but in the past 6 months odd things have happened.

While the server is never down—meaning it still responds to pings & even file sharing as well as the print server is active/usable—for undetermined reasons the VNC & Remote Desktop capabilities suddenly become unreachable from any system: Mac or Windows using VNC or Apple Remote Desktop doesn’t help.

What I have been able to do in the past is SSH in and restart the ARDAgent like this:

sudo /System/Library/CoreServices/RemoteManagement/ARDAgent.app/Contents/Resources/kickstart -restart -agent

Which cleared things up right away. But in the last week I suddenly can’t even get in via SSH.

I am 100% positive the machine is still alive & working since I kicked off a fairly huge data transfer process via AFP 2 weeks ago that is clearly still going since data is still being received by the remote machine.

So why can’t I get in via SSH or ARD/VNC? There is also a basic web page setup on the server I cannot reach as well via port 80 which worked fine before. Could the actual server software on the system have crashed? If so, why are the following ports still showing as active when running an nmap scan; hostname & IP changed for privacy but all output accurate:

nmap the_hostname_of_the_server -p0-8000
Starting Nmap 5.21 ( http://nmap.org ) at 2014-10-18 15:04 EDT
Nmap scan report for the_hostname_of_the_server (123.456.789.0)
Host is up (0.0031s latency).
rDNS record for 123.456.789.0: the_hostname_of_the_server
Not shown: 990 closed ports
PORT     STATE    SERVICE
22/tcp   open     ssh
25/tcp   open     smtp
80/tcp   open     http
88/tcp   open     kerberos-sec
139/tcp  open     netbios-ssn
311/tcp  open     asip-webadmin
445/tcp  open     microsoft-ds
548/tcp  open     afp
587/tcp  open     submission
625/tcp  open     apple-xsrvr-admin
631/tcp  open     ipp
5900/tcp filtered vnc

And here is the output of an SSH session with the -v option set for verbose output:

ssh -v jakegould@the_hostname_of_the_server 
OpenSSH_5.9p1 Debian-5ubuntu1.4, OpenSSL 1.0.1 14 Mar 2012
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
debug1: Connecting to the_hostname_of_the_server [123.456.789.0] port 22.
debug1: Connection established.
debug1: identity file /home/jakegould/.ssh/id_rsa type -1
debug1: identity file /home/jakegould/.ssh/id_rsa-cert type -1
debug1: identity file /home/jakegould/.ssh/id_dsa type -1
debug1: identity file /home/jakegould/.ssh/id_dsa-cert type -1
debug1: identity file /home/jakegould/.ssh/id_ecdsa type -1
debug1: identity file /home/jakegould/.ssh/id_ecdsa-cert type -1

That’s basically all that I get when I attempt to SSH into the server. It hangs and hangs and hangs forever until I force an exit via hitting control+c.

I am fairly certain rebooting this server after it deals with my file transfer will bring services back up, but is there anything else I could do to get services back up & running other than a reboot? Or do I have the choice of hard reboot or being onsite plugging in a keyboard & monitor to then force services back up on site?

To be completely fair, this server is pretty much on it’s way out and is only being used for boring utility transfers; behind the scenes stuff a Linux box could probably handle. The Finder crashes for odd reasons & there are times we must physically give the “one finger salute” to get it to work again. So I don’t want to spend too much time dealing with a “lame duck”, but it would be nice to know if there is something else that could be done to restore at least remote SSH connectivity.

The subject sums up the scenario. I believe the answer is there is no downtime but the system might simply be slower during the rebuild. Which is fine. But if there is downtime, how long? A small blip for a short maintenance window is acceptable. Days of rebuild would require a new plan to assess how to handle.

Details: The client has an iSCSI connection going to a Synology RS2414RP+ (running DSM 5) which is a rack mounted unit with 12 bays. All bays are populated with 3TB drives since it was initially setup. And since it is a device dedicated to one large iSCSI volume, it was setup to use the max amount of space. Which has worked for months despite that space being around 1GB in size.

Now the system has occasionally run into “read only” mode which I believe is being caused by the thin provisioning of the iSCSI volume reaching the outer limits of the volume itself. Managed to bring the volume “back to life” by SSHing in & tossing an old/side-by-side backup of a PostgreSQL database which freed up about 36MB of space on the underlying volume which holds the iSCSI partition. Which was enough breathing room to get the iSCSI volume to mount again. But obviously is a problem waiting to happen.

So, since this Synology unit has the volume formatted as a Synology Hybrid RAID setup with 2 disk fault tolerance, the decision was made to replace one of the 3TB drives with a 4TB drive which should expand the volume. The drive has been inserted, is formatted & is running through the slow as molasses parity check stage.

But once parity checking is done, I will have to login to expand the volume. At this final stage, what happens from a service outage perspective? From what I have read online since the Synology DSM is is based on standard open source tools such as LVM (Logical Volume Manager), the volume will expand without having to incur an outage. But since we are talking about 27TB+ of storage, I want to doubly/triply make sure my assumption is correct.

Which is basically: When the parity checking is done, and I go into expand the volume, all services will still be up and running & the expansion will occur as a background process, thus ensuring zero downtime?

Okay, an Ubuntu server I manage fell victim to a DDoS attack today. Usually this is unpleasant but not that big a deal. A few high server load moments & then it passes. Today was clearly different. For the record, I have years of server attack scars. But today felt different. Read on.

The solution was solved by re-enabling a ModSecurity rule that prevents curl access via a call to a pure IP address. Meaning if a hostname is mygreatsite.com and the IP address is 123.456.78.90 and I have a virtual host for mygreatsite.com & that is enabled. And ModSecurity is active with the rule that prevents curl access via a call to a pure IP address, then a visitor would get the proper content when they visit mygreatsite.com, but are blocked by ModSecurity with a 403: Forbidden if they attempt to access the site via 123.456.78.90. Great problem solved!

But while the attack was happening, three things were clear:

1. Apache Processes Through the Roof: The Apache process load as reflected in Munin was through the roof & consistently 3x higher than normal.
2. No Corresponding Web Traffic Logged: There was no corresponding traffic logged via Apache & reflected in AWStats. This was the case for a few hours. This is across all logs on all virtual hosts on the server including the default virtual host that would be connected to a bare IP address.
3. Server Load Was Moderately High, But Not Typical Attack Level: While overall server load was high, it was only in the single digits—4, 5, 6, 7, etc…—and not a typical DDoS where loads can go double and even triple digit like 20, 30 & even 120 (!!!).

To me item number 3 was the most unusual part of this attack. I have not experienced a DDoS where server load was basically equivalent of a heavy traffic day & not an insane server load connected to the attack.

Additionally, running elinks to fetch Apache server-status from the command line showed the connection pool filling up like crazy within minutes of a restart:

elinks http://localhost/server-status?refresh=1

And past any of that, I just noticed an Ubuntu security patch today specifically for curl issues. And this specific issue piqued my interest:

libcurl incorrectly validates wildcard SSL certificates containing literal IP addresses.

So here is the question: What the heck happened? I know re-enabling a ModSecurity rule stopped the attack, but could it have in any way been connected to the libcurl issue? And if so, how can I detect or prove that? And past that if the issue was with libcurl—which I assume is coming from weaponized clients using the flaw—then does that mean that someone can simple keep a hacked older version of curl around to DDoS again? And writing that, any hacker can just download source code & patch it to hack so why the flood today?

My guess is the patch to curl made more people aware of the flaw & the “enthusiasm” of script kiddies around the globe brought a mad flood of scanning around within hours of the exploit being found? But again, this confuses me because it’s on the client curl install, correct? I can’t imagine curl on the server side would be victim to this.

Working on a site where the plan is to move URLs from a query string format to a number based format. Lots of URLs exist that have unescaped accented & similar UTF8 characters in them. The problem? I can’t seem to get Apache2 to properly match accented characters & do a rewrite. I am doing this all in the Apache2 config.

For example, this URL:

http://great.website.example.com/?place=cafe

Will work as expected with this Apache2 RewriteRule setting:

  RewriteCond %{QUERY_STRING} ^(place|location)=cafe
  RewriteRule ^/find/$ /find/1234? [L,R=301]

Now look at this URL. Note the accented é:

http://great.website.example.com/?place=café

Why doesn’t that URL work with the following Apache2 RewriteRule setting:

  RewriteCond %{QUERY_STRING} ^(place|location)=café
  RewriteRule ^/find/$ /find/1234? [L,R=301]

Both of these rules should rewrite the URL to the following:

http://great.website.example.com/find/1234

But the example with the accented é simply doesn’t work. Maybe a wildcard character would work, but I can’t seem to get that to work either.

The subject mostly says it all. I am in charge of a few web servers running Ubuntu 12.04 running Apache2 & I would like to setup APC.

Now I understand APC can hit segmentation fault issues when acting on PHP code that has errors or quirks. So cleanup is advisable, but not really practical for a manpower standpoint.

So I have cooked up a script to monitor the main Apache2 error.log & count how many segmentation faults it sees. I run it as a cron job that runs every 2 minutes. If it hits a specified number of segmentation faults, it should automatically restart Apache2 to clear out APC & get services running smoothly again. The seed idea of this script comes from a comment on this page but I have really heavily built upon the concept to make it more production ready for my needs & tastes.

I am generally happy with this script, but feel one major improvement is the core logic in this snippet:

if [[ `tail -n ${TAIL_NUMLINES} "${APACHE_ERROR_LOG}" | egrep -c "${TEXT_TO_WATCH}"` -ge ${FAIL_COUNT} ]]; then

That is basically the core logic that tails the log file for a specified number of lines, and if it sees a specified number of log entries with exit signal Segmentation fault in them, then it decides it's time to do something. In this case, log the incident, e-mail someone about the incident & restart Apache.

I would like that logic to factor in time because errors can be few & far between. So there are cases where my FAIL_COUNT simply matches the TAIL_NUMLINES despite a restart because no new entries are in the main error log because no new errors have hit. Which would end up in a situation where the server basically restarts every time the cron job hits. Which is horrible.

So my stop-gap solution for now is to set FAIL_COUNT & TAIL_NUMLINES to a low enough number to match the standard counts of entries that are created when Apache2 reloads. But I still don't like that.

So what—if anything—can be done to add a time frame to my tail/egrep logic. Also, I would like to avoid creating a timestamp or log line position hint being saved to a file if possible. I want this script to be self-contained other than depending on the cron job.

Full script as I have it now.

#!/bin/bash

LOCK_NAME="APACHE_LOGWATCHER"
LOCK_DIR=/tmp/${LOCK_NAME}.lock
PID_FILE=${LOCK_DIR}/${LOCK_NAME}.pid

DATE=`date +%Y%m%d`
TIME=`date +%H%M`
# SUFFIX="-"${DATE}"-"${TIME};
SUFFIX="-"${DATE};

APACHE_ERROR_LOG="/var/log/apache2/error.log"
APACHE_RESTART="/etc/init.d/apache2 restart"
TEXT_TO_WATCH="exit signal Segmentation fault"

HOSTNAME=$(hostname)
MAIL_ADDRESS="[email protected]"
MAIL_SUBJECT=${HOSTNAME}": Apache Segfault Notification"

SCRIPT_NAME=$(basename "$0")
SCRIPT_BASE_NAME=${SCRIPT_NAME%.*}

LOG_DIR="/opt/segfault_logs/"
LOG_FILENAME=${SCRIPT_BASE_NAME}${SUFFIX}".log"
LOG_FULLPATH=${LOG_DIR}${LOG_FILENAME}

TAIL_NUMLINES=5
FAIL_COUNT=4

# If the Apache log file doesn't exist, then exit.
if [ ! -f ${APACHE_ERROR_LOG} ]; then
    exit
fi

# Main process.
if mkdir ${LOCK_DIR} 2>/dev/null; then
    # If the ${LOCK_DIR} doesn't exist, then start working & store the ${PID_FILE}
    echo $$ > ${PID_FILE}

    STARTUP_MESSAGE="`date` Log watcher starting."
    if [ -d ${LOG_DIR} ]; then
        echo ${STARTUP_MESSAGE} >> ${LOG_FULLPATH}
    fi

    # Tail--but do not follow--a chunk of the LOG_FULLPATH if the number of instances is
    # greater than or equal to the FAIL_COUNT, act
    if [[ `tail -n ${TAIL_NUMLINES} "${APACHE_ERROR_LOG}" | egrep -c "${TEXT_TO_WATCH}"` -ge ${FAIL_COUNT} ]]; then

        # Create the log message.
        LOG_MESSAGE="`date` Segfault detected on "$HOSTNAME

        # Log the error to the file.
        if [ -d ${LOG_DIR} ]; then
            echo ${LOG_MESSAGE} >> ${LOG_FULLPATH}
        fi

        # Send e-mail notification.
        echo ${LOG_MESSAGE}$'\n\r'${FAIL_COUNT} | mail -s "${MAIL_SUBJECT}" ${MAIL_ADDRESS}

        # Restart Apache
        ${APACHE_RESTART}
    fi

    rm -rf ${LOCK_DIR}
    exit
else
    if [ -f ${PID_FILE} ] && kill -0 $(cat ${PID_FILE}) 2>/dev/null; then
        # Confirm that the process file exists & a process
        # with that PID is truly running.
        # echo "Running [PID "$(cat ${PID_FILE})"]" >&2
        exit
    else
        # If the process is not running, yet there is a PID file--like in the case
        # of a crash or sudden reboot--then get rid of the ${LOCK_DIR}
        rm -rf ${LOCK_DIR}
        exit
    fi
fi

EDIT: Here is an example of the Apache2 logfile output the above script is monitoring.

[Sat Mar 02 14:32:26 2013] [notice] child pid 14696 exit signal Segmentation fault (11)
[Sat Mar 02 14:32:27 2013] [notice] child pid 13914 exit signal Segmentation fault (11)
[Sat Mar 02 14:32:27 2013] [notice] child pid 15735 exit signal Segmentation fault (11)
[Sat Mar 02 14:32:28 2013] [notice] child pid 14865 exit signal Segmentation fault (11)
[Sat Mar 02 14:32:28 2013] [notice] child pid 15545 exit signal Segmentation fault (11)
[Sat Mar 02 14:32:30 2013] [notice] child pid 13821 exit signal Segmentation fault (11)
[Sat Mar 02 14:32:31 2013] [notice] child pid 15683 exit signal Segmentation fault (11)
[Sat Mar 02 14:32:47 2013] [notice] child pid 15684 exit signal Segmentation fault (11)
[Sat Mar 02 14:33:54 2013] [notice] child pid 15482 exit signal Segmentation fault (11)
[Sat Mar 02 14:34:04 2013] [notice] caught SIGTERM, shutting down
[Sat Mar 02 14:34:06 2013] [notice] ModSecurity for Apache/2.6.3 (http://www.modsecurity.org/) configured.
[Sat Mar 02 14:34:06 2013] [notice] ModSecurity: APR compiled version="1.4.6"; loaded version="1.4.6"
[Sat Mar 02 14:34:06 2013] [notice] ModSecurity: PCRE compiled version="8.12"; loaded version="8.12 2011-01-15"
[Sat Mar 02 14:34:06 2013] [notice] ModSecurity: LUA compiled version="Lua 5.1"
[Sat Mar 02 14:34:06 2013] [notice] ModSecurity: LIBXML compiled version="2.7.8"
[Sat Mar 02 14:34:07 2013] [notice] Apache/2.2.22 (Ubuntu) mod_ssl/2.2.22 OpenSSL/1.0.1 configured -- resuming normal operations

Very bizarre situation. I have inherited a few Xservers from a previous admin that I have been charged with cleaning up on various levels. I am familiar with settings in Ubuntu and am very comfortable with the command line—including compiling from source—on top of knowing how to maneuver around the command line in OS X as well. But have come across something utterly baffling on one of the servers which is running 10.6.8 (Snow Leopard).

First oddity, I would like to turn the software firewall completely off. Sounds as simple as going into Server Admin and just telling it to stop the firewall right? Nope! So if I turn off the firewall, and I do the following command to double check in the command line:

sysctl -a | grep net.inet.ip.fw.enable

The results are as expected:

net.inet.ip.fw.enable: 0

But then if I check it again within 5 minutes or so, it becomes magically enabled again:

net.inet.ip.fw.enable: 1

Ditto with NAT forwarding.

sysctl -a | grep net.inet.ip.forwarding

Disable it in the Server Admin, and this is the result:

net.inet.ip.forwarding: 0

A few minutes later:

net.inet.ip.forwarding: 1

WTF?!? I checked crontabs and I am the only user on the machine.

I do have the Server Admin app on another machine set to watch this machine, but it’s not running all of the time. And this issue did exist prior to that. I am not 10o% familiar with sysctl so need some guidance. FWIW, I do have some Ethernet tweaks I have set in /etc/sysctl.conf very recently, but no other commands or settings connected to the above.

Why would these settings seem to magically change after a few minutes? What can be done to stop that.

Thanks in advance for the help & pointers.

EDIT: Contents of /etc/hostconfig as per one of the comments below:

AFPSERVER=-NO-
AUTHSERVER=-NO-
TIMESYNC=-NO-
QTSSWEBADMIN=-NO-
QTSSRUNSERVER=-NO-
MYSQLCOM=-YES-
IPFORWARDING=-NO-