Sammitch's questions

I've been tasked with setting up longer/better log retention for our postgres servers, but after configuring postgres to ship logs to syslog via local0 I've found that they never actually hit the configured local file.

/etc/rsyslog.d/postgresql.conf:

local0.* /company/data/psql/company_cluster/log/postgresql.log

Relevant postgres config:

log_destination = 'syslog'
syslog_facility = 'local0'

Once I got the config in place and reloaded/restarted services I noticed that while nothing was going into the specified log file [even after touching it] I was seeing messages piped into graylog which has a *.* rule.

Even stranger is that logger -p local0.info test or any other facility doesn't generate a message anywhere that I can find. Same goes for any other local facility, or even mail.none.

Rsyslog seems to be restarting cleanly, I've doubled-checked the selinux context on the config file, and I can't find any relevant info logged from rsyslog itself. What's happening?

All rsyslog config:

# cat /etc/rsyslog.conf /etc/rsyslog.d/* | grep -v -e '^\s*#' -e '^\s*$'
$ModLoad imuxsock
$ModLoad imjournal
$WorkDirectory /var/lib/rsyslog
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
$IncludeConfig /etc/rsyslog.d/*.conf
$OmitLocalLogging on
$IMJournalStateFile imjournal.state
*.info;mail.none;authpriv.none;cron.none                /var/log/messages
authpriv.*                                              /var/log/secure
mail.*                                                  -/var/log/maillog
cron.*                                                  /var/log/cron
*.emerg                                                 :omusrmsg:*
uucp,news.crit                                          /var/log/spooler
local7.*                                                /var/log/boot.log

# graylog.conf
*.* @graylogmaster.rancher.company.ca:8514;RSYSLOG_SyslogProtocol23Format
# listen.conf
$SystemLogSocketName /run/systemd/journal/syslog
# postgresql.conf
local0.* /company/data/psql/company_cluster/log/postgresql.log 

Edit: "Not Really A Solution" Solution

Something seems to be wrong with Rsyslog [and maybe more] on the particular server that I was working on when I originally posted this question. Restarting the service hasn't done anything and I'm waiting for a maintenance window to either restart or reprovision this server.

This same config works just fine on other servers, but with the caveat that LOCAL0 seems to be invalid in at least some contexts, but local0 works everywhere.

I've got a chef recipe to bootstrap some worker nodes that I've been using for quite some time, and for the first time in a while I needed to increase the size of my worker pool, but found that the new nodes were not bootstrapping properly due to the yum error:

Error: Package: cyrus-sasl-md5-2.1.26-21.el7.x86_64 (abc-os)
           Requires: cyrus-sasl-lib(x86-64) = 2.1.26-21.el7
           Installed: cyrus-sasl-lib-2.1.26-20.el7_2.x86_64 (@abc-os)
               cyrus-sasl-lib(x86-64) = 2.1.26-20.el7_2

Where abc-os is our internal repo so we can lock down package versions. Naturally I assumed that this was the problem, but on checking into the repo I found that the complete set of packages for both 21.el7 and 20.el7_2 are present.

Furthermore running yum list available cyrus-sasl-md5 gives:

cyrus-sasl-md5.i686       2.1.26-20.el7_2    abc-os
cyrus-sasl-md5.x86_64     2.1.26-21.el7      abc-os

And querying the repo's sqlite file in /var/cache/yum/x86_64/7/abc-os/gen returns:

sqlite> SELECT name, arch, version, epoch, release FROM packages WHERE name LIKE 'cyrus%' ORDER BY arch, version, epoch, release, name;
cyrus-sasl-lib  i686        2.1.26      0           17.el7
cyrus-sasl-lib  i686        2.1.26      0           20.el7_2
cyrus-sasl-md5  i686        2.1.26      0           20.el7_2
cyrus-sasl-pla  i686        2.1.26      0           20.el7_2
cyrus-sasl      x86_64      2.1.26      0           17.el7
cyrus-sasl-dev  x86_64      2.1.26      0           17.el7
cyrus-sasl-lib  x86_64      2.1.26      0           17.el7
cyrus-sasl      x86_64      2.1.26      0           20.el7_2
cyrus-sasl-dev  x86_64      2.1.26      0           20.el7_2
cyrus-sasl-lib  x86_64      2.1.26      0           20.el7_2
cyrus-sasl-md5  x86_64      2.1.26      0           20.el7_2
cyrus-sasl-pla  x86_64      2.1.26      0           20.el7_2
cyrus-sasl      x86_64      2.1.26      0           21.el7
cyrus-sasl-dev  x86_64      2.1.26      0           21.el7
cyrus-sasl-lib  x86_64      2.1.26      0           21.el7
cyrus-sasl-md5  x86_64      2.1.26      0           21.el7
cyrus-sasl-pla  x86_64      2.1.26      0           21.el7

At the moment I'm stumped as to why this is happening, and what to do to dig into this further.

For reference, the OS is CentOS 7.2 x86_64.

I'm writing some automations and I've run into a problem where a host's key has changed and a wrench gets thrown into the works. I'd like to add a pre-flight check for this specific condition, but I can't seem to get ssh to return anything more than its catch-all 255 return code.

eg:

$ ssh -o StrictHostKeyChecking=yes foobar@squiffy-host ls; echo $?
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@       WARNING: POSSIBLE DNS SPOOFING DETECTED!          @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
...snip...
255

How can I detect this one, specific case?

I'm trying to stand up a syslog server to receive events from a piece of network equipment, but I can't seem to get it to actually log the events to disk. I can confirm that the connection is being made successfully, and that rsyslogd is getting the event, I just can't figure out why it's not doing anything with it. The only thing that gets logged is the startup of rsyslogd itself.

Config and debug outputs below, this was originally generated from the rsyslog chef cookbook, but I added a couple tweaks to fix the super deprecated syntax that it was complaining about.

rsyslog.conf:

$MaxMessageSize 2k
$PreserveFQDN off
$ModLoad imuxsock
$ModLoad imklog
$ModLoad imtcp
$InputTCPServerRun 514
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
$RepeatedMsgReduction on
$WorkDirectory /var/spool/rsyslog
$FileOwner root
$FileGroup root
$FileCreateMode 0640
$DirCreateMode 0755
$Umask 0022
$IncludeConfig /etc/rsyslog.d/*.conf

rsyslog.d/*:

$DirGroup root
$DirCreateMode 0755
$FileGroup root
$template PerHostAuth,"/var/log/rsyslog/%HOSTNAME%/auth.log"
$template PerHostCron,"/var/log/rsyslog/%HOSTNAME%/cron.log"
$template PerHostSyslog,"/var/log/rsyslog/%HOSTNAME%/syslog"
$template PerHostDaemon,"/var/log/rsyslog/%HOSTNAME%/daemon.log"
$template PerHostKern,"/var/log/rsyslog/%HOSTNAME%/kern.log"
$template PerHostLpr,"/var/log/rsyslog/%HOSTNAME%/lpr.log"
$template PerHostUser,"/var/log/rsyslog/%HOSTNAME%/user.log"
$template PerHostMail,"/var/log/rsyslog/%HOSTNAME%/mail.log"
$template PerHostMailInfo,"/var/log/rsyslog/%HOSTNAME%/mail.info"
$template PerHostMailWarn,"/var/log/rsyslog/%HOSTNAME%/mail.warn"
$template PerHostMailErr,"/var/log/rsyslog/%HOSTNAME%/mail.err"
$template PerHostNewsCrit,"/var/log/rsyslog/%HOSTNAME%/news.crit"
$template PerHostNewsErr,"/var/log/rsyslog/%HOSTNAME%/news.err"
$template PerHostNewsNotice,"/var/log/rsyslog/%HOSTNAME%/news.notice"
$template PerHostDebug,"/var/log/rsyslog/%HOSTNAME%/debug"
$template PerHostMessages,"/var/log/rsyslog/%HOSTNAME%/messages"
auth,authpriv.*         ?PerHostAuth
*.*;auth,authpriv.none  -?PerHostSyslog
cron.*                  ?PerHostCron
daemon.*                -?PerHostDaemon
kern.*                  -?PerHostKern
lpr.*                   -?PerHostLpr
mail.*                  -?PerHostMail
user.*                  -?PerHostUser
mail.info               -?PerHostMailInfo
mail.warn               ?PerHostMailWarn
mail.err                ?PerHostMailErr
news.crit               ?PerHostNewsCrit
news.err                ?PerHostNewsErr
news.notice             -?PerHostNewsNotice
*.=debug;\
  auth,authpriv.none;\
  news.none;mail.none   -?PerHostDebug
*.=info;*.=notice;*.=warn;\
  auth,authpriv.none;\
  cron,daemon.none;\
  mail,news.none        -?PerHostMessages
:fromhost-ip,!isequal,"127.0.0.1" stop
*.info;mail.none;authpriv.none;cron.none    /var/log/messages
authpriv.*    /var/log/secure
mail.*    -/var/log/maillog
cron.*    /var/log/cron
*.emerg    :omusrmsg:*
uucp,news.crit    /var/log/spooler
local7.*    /var/log/boot.log
$SystemLogSocketName /run/systemd/journal/syslog

Test event:

logger --server 127.0.0.1 --tcp --port 514 -p user.info test

strace -f of rsyslogd receiving the event, but doing nothing:

[pid 32310] accept(5, {sa_family=AF_INET, sin_port=htons(59755), sin_addr=inet_addr("127.0.0.1")}, [16]) = 11
[pid 32310] rt_sigprocmask(SIG_BLOCK, [HUP], ~[KILL STOP TTIN RTMIN RT_1], 8) = 0
[pid 32310] open("/etc/hosts", O_RDONLY|O_CLOEXEC) = 12
[pid 32310] fstat(12, {st_mode=S_IFREG|0644, st_size=451, ...}) = 0
[pid 32310] mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fb290d18000
[pid 32310] read(12, "#\n# This file is managed by Chef"..., 4096) = 451
[pid 32310] close(12)                   = 0
[pid 32310] munmap(0x7fb290d18000, 4096) = 0
[pid 32310] rt_sigprocmask(SIG_SETMASK, ~[KILL STOP TTIN RTMIN RT_1], NULL, 8) = 0
[pid 32310] fcntl(11, F_GETFL)          = 0x2 (flags O_RDWR)
[pid 32310] fcntl(11, F_SETFL, O_RDWR|O_NONBLOCK) = 0
[pid 32310] epoll_ctl(10, EPOLL_CTL_ADD, 11, {EPOLLIN, {u32=2147497424, u64=140404628403664}}) = 0
[pid 32310] epoll_wait(10, {{EPOLLIN, {u32=2147497424, u64=140404628403664}}}, 128, -1) = 1
[pid 32310] recvfrom(11, "<14>Jul  7 22:41:54 stack: test\0", 131072, MSG_DONTWAIT, NULL, NULL) = 32
[pid 32310] gettimeofday({1467931314, 137777}, NULL) = 0
[pid 32310] epoll_wait(10, {{EPOLLIN, {u32=2147497424, u64=140404628403664}}}, 128, -1) = 1
[pid 32310] recvfrom(11, "", 131072, MSG_DONTWAIT, NULL, NULL) = 0
[pid 32310] epoll_ctl(10, EPOLL_CTL_DEL, 11, {EPOLLIN, {u32=2147497424, u64=140404628403664}}) = 0
[pid 32310] close(11)                   = 0
[pid 32310] epoll_wait(10,

I've got a cluster of machines running Carbon and Graphite that I need to scale for more storage, but I'm not sure if I need to scale up or out.

The cluster is currently comprised of:

• 1 Relay Node: Receives all metrics and forwards to the relevant storage node
• 6 Storage Nodes: Houses all the Whisper DB files

The problem is that it seems like when the disks got in the neighbourhood of 80% usage the performance fell off of a cliff. Cluster write IOPS fell from a near-constant 13k to a more chaotic average of around 7k and IOwait time averages 54%.

I've had a look through our config repo and there are no changes since early April, so this isn't the result of a config change.

Question: Will increasing the disk size bring IO performance back under control, or do I need to add more storage nodes?

Note: No SSDs here, just lots and lots of spindles.

Relevant Graphs:

Stats and Stuff:

e2freefrag:

[root@graphite-storage-01 ~]# e2freefrag /dev/vda3
Device: /dev/vda3
Blocksize: 4096 bytes
Total blocks: 9961176
Free blocks: 4781849 (48.0%)

Min. free extent: 4 KB
Max. free extent: 81308 KB
Avg. free extent: 284 KB
Num. free extent: 19071

HISTOGRAM OF FREE EXTENT SIZES:
Extent Size Range :  Free extents   Free Blocks  Percent
    4K...    8K-  :          4008          4008    0.08%
    8K...   16K-  :          1723          3992    0.08%
   16K...   32K-  :           703          3495    0.07%
   32K...   64K-  :           637          7400    0.15%
   64K...  128K-  :          1590         29273    0.61%
  128K...  256K-  :          4711        236839    4.95%
  256K...  512K-  :          2664        265691    5.56%
  512K... 1024K-  :          2359        434427    9.08%
    1M...    2M-  :           595        213173    4.46%
    2M...    4M-  :            75         49182    1.03%
   64M...  128M-  :             6        118890    2.49%

e4defrag:

[root@graphite-storage-01 ~]# e4defrag -c /dev/vda3
<Fragmented files>                             now/best       size/ext
1. /opt/graphite/storage/graphite.db            17/1              4 KB
2. /var/log/cron                                13/1              4 KB
3. /var/log/wtmp                                16/1              4 KB
4. /root/.bash_history                           4/1              4 KB
5. /var/lib/rpm/Sha1header                      10/1              4 KB

 Total/best extents                             182256/159981
 Average size per extent                        183 KB
 Fragmentation score                            2
 [0-30 no problem: 31-55 a little bit fragmented: 56- needs defrag]
 This device (/dev/vda3) does not need defragmentation.
 Done.

iostat:

[root@graphite-storage-01 ~]# iostat -k -x 60 3
Linux 3.10.0-229.7.2.el7.x86_64 (graphite-storage-01)     07/05/2016      _x86_64_        (2 CPU)

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           7.99    0.00    2.54   29.66    0.35   59.46

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
vda               0.00   100.34  177.48 1808.94  2715.66  7659.19    10.45     0.26    0.13    0.65    0.08   0.23  46.14

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           6.17    0.00    7.00   73.21    0.58   13.04

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
vda               0.00    23.87  672.40  656.47  8729.87  2752.27    17.28     7.36    5.50    2.72    8.35   0.73  96.83

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           7.06    0.00    7.31   73.03    0.59   12.01

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
vda               0.00    42.68  677.67  614.88  8634.93  2647.53    17.46     6.66    5.15    2.72    7.83   0.74  96.08

df:

[root@graphite-storage-01 ~]# df
Filesystem     1K-blocks     Used Available Use% Mounted on
/dev/vda3       39153856 33689468   3822852  90% /
devtmpfs         1933092        0   1933092   0% /dev
tmpfs            1941380        0   1941380   0% /dev/shm
tmpfs            1941380   188700   1752680  10% /run
tmpfs            1941380        0   1941380   0% /sys/fs/cgroup
/dev/vda2         999320     2584    980352   1% /tmp
[root@graphite-storage-01 ~]# df -i
Filesystem      Inodes  IUsed   IFree IUse% Mounted on
/dev/vda3      2490368 239389 2250979   10% /
devtmpfs        483273    304  482969    1% /dev
tmpfs           485345      1  485344    1% /dev/shm
tmpfs           485345    322  485023    1% /run
tmpfs           485345     13  485332    1% /sys/fs/cgroup
/dev/vda2        65536     22   65514    1% /tmp

Edit: I've resized one of the storage nodes, but it's not had an effect. I've also found the cachestat utility in [https://github.com/brendangregg/perf-tools](a collection of perf tools) that's given me a look inside the VFS cache. At this point it looks like I've reached the limit on the IO throughput that my storage can provide.

At this point I think I'm either going to have to continue to scale out to more cluster members, or see about finding a more write-efficient time-series storage solution.

Example output from cachestat:

storage-01 [resized disk]
    HITS   MISSES  DIRTIES    RATIO   BUFFERS_MB   CACHE_MB
    9691    14566     7821    40.0%          160       2628
   36181    14689     7802    71.1%          160       2631
    8649    13617     7003    38.8%          159       2628
   15567    13399     6857    53.7%          160       2627
    9045    14002     7049    39.2%          160       2627
    7533    12503     6153    37.6%          159       2620

storage-02 [not resized]
    HITS   MISSES  DIRTIES    RATIO   BUFFERS_MB   CACHE_MB
    5097    11629     4740    30.5%          143       2365
    5977    11045     4843    35.1%          142       2344
    4356    10479     4199    29.4%          143       2364
    6611    11188     4946    37.1%          143       2348
   33734    14511     5930    69.9%          143       2347
    7885    16353     7090    32.5%          143       2358

Super Late Edit: We've since migrated to another platform where SSDs are available and, while things were good for some time, we eventually saw the same sharp decline in performance as we added more and more metrics. While I don't have any definitive proof I believe that this is a corner case between how Carbon/Whisper storage works, and the sheer number of metrics we store.

Basically, so long as the system has enough RAM to comfortably cache the Whisper files for reads the IO is almost pure write and everything is happy. However, once FS cache starvation sets in and Whisper files need to be continually read in off disk that eats into your IO bandwidth and everything starts going to pot.

I've got a few servers that have begun oom-killing their backup processes and, while I understand that encountering the oom condition is quite bad in itself, I need this process to not die so that backups happen properly while the memory issue is addressed.

To that end I've attempted to create way to launch processes with adjusted oom_scores in a way similar to launching a process with nice.

#!/bin/bash

function oom_adj_exec() {
    while getopts ':n:' opt; do
        case $opt in
            n)
                if grep -q '^-\?[0-9]\+$' <(echo "$OPTARG"); then
                    if [ "$OPTARG" -ge -1000 -a "$OPTARG" -le 1000 ]; then
                        oom_score_adjust=$OPTARG
                    else
                        echo "Acceptable values for -n are from -1000 to 1000" >&2
                        return 255
                    fi
                else
                    echo "Improper format for -n: $OPTARG" >&2
                    return 255
                fi
                break
                ;;
            :)
                echo "option -$OPTARG requires a value" >&2
                return 255
                ;;
            *)
                echo "Unknown option -$opt" >&2
                return 255
                ;;
        esac
    done

    command=${@:$OPTIND}

    # job control requires the monitor option which
    # is usually not set for non-interactive shells
    prev_state=$(set +o | grep monitor)
    set -o monitor

    $command &
    pid=$!

    echo "$oom_score_adjust" > /proc/$pid/oom_score_adj

    fg %% > /dev/null

    ecode=$?

    # restore the previous state of the shell
    $prev_state

    return $ecode
}

oom_adj_exec $@

Example usage:

./oom_adj_exec.sh -n -500 /usr/bin/mem_bloater

While it seems to work I can't shake the feeling like there's something waiting in there to go horribly wrong. Is there anything that stands out as being a truly terrible idea and/or disaster waiting to happen?

I've got an old master nameserver running Bind 9.2 and a newer slave running 9.8. Right now we've got a project going on where we're essentially splitting a cloud in two and we're using subzones and CNAMEs to keep our services running smoothly. However, the cranky old 9.2 server doesn't seem to want to resolve the CNAMEs to the subzone and returns REFUSED: recursion requested but not available. On the other hand the 9.8 server serves the requests just fine.

Disclaimer: I know these nameservers are horribly out of date, and even worse the one running 9.2's OS is waaaaay out of support as well, so I'm not likely to find a reputable package to upgrade it. The project immediately after this cloud split is rebuilding our DNS servers/services from scratch.

How can I get the older server to resolve these CNAMEs properly?

dig results

dig @ NS1 [Bind 9.2]

# dig foo.domain.com @ns1.domain.com

; <<>> DiG 9.8.2rc1-RedHat-9.8.2-0.30.rc1.el6_6.3 <<>> foo.domain.com @ns1.domain.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: REFUSED, id: 5937
;; flags: qr rd; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0
;; WARNING: recursion requested but not available

;; QUESTION SECTION:
;foo.domain.com.        IN      A

;; Query time: 116 msec
;; SERVER: 4.3.2.1#53(4.3.2.1)
;; WHEN: Fri Jul 31 16:18:36 2015
;; MSG SIZE  rcvd: 48

dig @ NS2 [Bind 9.8]

# dig foo.domain.com @ns2.domain.com

; <<>> DiG 9.8.2rc1-RedHat-9.8.2-0.30.rc1.el6_6.3 <<>> foo.domain.com @ns2.domain.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 59986
;; flags: qr aa rd; QUERY: 1, ANSWER: 2, AUTHORITY: 2, ADDITIONAL: 2
;; WARNING: recursion requested but not available

;; QUESTION SECTION:
;foo.domain.com.        IN      A

;; ANSWER SECTION:
foo.domain.com. 300 IN  CNAME   foo.sub.domain.com.
foo.sub.domain.com. 300 IN A     5.6.7.8

;; AUTHORITY SECTION:
sub.domain.com.  300     IN      NS      ns1.domain.com.
sub.domain.com.  300     IN      NS      ns2.domain.com.

;; ADDITIONAL SECTION:
ns1.domain.com.     30      IN      A       4.3.2.1
ns2.domain.com.     30      IN      A       1.2.3.4

;; Query time: 80 msec
;; SERVER: 1.2.3.4#53(1.2.3.4)
;; WHEN: Fri Jul 31 16:22:29 2015
;; MSG SIZE  rcvd: 161

Config

Below are the config files for the servers, trimmed down to the bare essentials.

NS1 [Bind 9.2]

options {
        recursion no;
        additional-from-auth no;
        additional-from-cache no;
        blackhole { bogon; };
        directory "/var/named";
        notify yes;
};
zone "domain.com" {
        type master;
        file "/var/named/domain.com.hosts";
        also-notify { 1.2.3.4; };
        notify yes;
};
zone "sub.domain.com" {
        type master;
        file "/var/named/sub.domain.com.hosts";
        also-notify { 1.2.3.4; };
        notify yes;
};

NS2 [Bind 9.8]

options {
        directory       "/var/named";
        recursion no;
        blackhole{ bogon; };
        dnssec-enable yes;
        dnssec-validation yes;
        dnssec-lookaside auto;
};
zone "domain.com" {
        type slave;
        masters { 4.3.2.1; };
        allow-transfer { 4.3.2.1; };
        file "/var/named/slaves/domain.com.hosts";
};
zone "sub.domain.com" {
        type slave;
        masters { 4.3.2.1; };
        allow-transfer { 4.3.2.1; };
        file "/var/named/slaves/sub.domain.com.hosts";
};

domain.com.hosts

$ORIGIN .
$TTL 300        ; 5 minutes
domain.com      IN SOA  ns1.domain.com. servers.domain.com. ( ... )
    NS      ns1.domain.com.
    NS      ns2.domain.com.
$ORIGIN domain.com.
sub NS ns1.domain.com.
sub NS ns2.domain.com.
foo CNAME foo.sub

sub.domain.com.hosts

$ORIGIN .
$TTL 300        ; 5 minutes
sub.domain.com   IN SOA  ns1.domain.com. servers.domain.com. ( ... )
    NS      ns1.domain.com.
    NS      ns2.domain.com.
$ORIGIN sub.domain.com.
foo A 5.6.7.8

So we've got two offices connected via Sonicwall IKE VPN:

• HQ is 10.42.0.0/16
• Remote is 10.63.0.0/16

There is an MS file share on a Windows 7 Pro box in HQ, 10.42.3.203, and machines in the Remote office need to access it. Both firewalls have 'allow any to any' rules for traffic between the two networks, and no deny rules that would apply to the traffic.

Below is a tshark transcript of someone in the remote office attempting to access the share in HQ. The machine running it listens on a mirrored uplink port behind the HQ firewall.

414.411940   10.63.3.39 -> 10.42.3.203  TCP 66 55628 > microsoft-ds [SYN] Seq=0 Win=8192 Len=0 MSS=1398 WS=256 SACK
415.518100   10.63.3.39 -> 10.42.3.203  TCP 66 55635 > microsoft-ds [SYN] Seq=0 Win=8192 Len=0 MSS=1398 WS=256 SACK
415.519325   10.63.3.39 -> 10.42.3.203  TCP 66 55636 > netbios-ssn [SYN] Seq=0 Win=8192 Len=0 MSS=1398 WS=256 SACK_
417.429670   10.63.3.39 -> 10.42.3.203  TCP 66 [TCP Retransmission] 55628 > microsoft-ds [SYN] Seq=0 Win=8192 Len=0
418.516965   10.63.3.39 -> 10.42.3.203  TCP 66 [TCP Retransmission] 55636 > netbios-ssn [SYN] Seq=0 Win=8192 Len=0
418.516969   10.63.3.39 -> 10.42.3.203  TCP 66 [TCP Retransmission] 55635 > microsoft-ds [SYN] Seq=0 Win=8192 Len=0
423.421594   10.63.3.39 -> 10.42.3.203  TCP 62 [TCP Retransmission] 55628 > microsoft-ds [SYN] Seq=0 Win=65535 Len=
424.525998   10.63.3.39 -> 10.42.3.203  TCP 62 [TCP Retransmission] 55636 > netbios-ssn [SYN] Seq=0 Win=8192 Len=0
424.526002   10.63.3.39 -> 10.42.3.203  TCP 62 [TCP Retransmission] 55635 > microsoft-ds [SYN] Seq=0 Win=8192 Len=0
436.553750   10.63.3.39 -> 10.42.3.203  NBNS 92 Name query NBSTAT *<00><00><00><00><00><00><00><00><00><00><00><00>
436.554051  10.42.3.203 -> 10.63.3.39   NBNS 217 Name query response NBSTAT
436.603070   10.63.3.39 -> 10.42.3.203  TCP 66 55690 > netbios-ssn [SYN] Seq=0 Win=8192 Len=0 MSS=1398 WS=256 SACK_
439.614949   10.63.3.39 -> 10.42.3.203  TCP 66 [TCP Retransmission] 55690 > netbios-ssn [SYN] Seq=0 Win=8192 Len=0
445.600591   10.63.3.39 -> 10.42.3.203  TCP 62 [TCP Retransmission] 55690 > netbios-ssn [SYN] Seq=0 Win=8192 Len=0
457.620875   10.63.3.39 -> 10.42.3.203  TCP 66 55734 > http [SYN] Seq=0 Win=8192 Len=0 MSS=1398 WS=4 SACK_PERM=1
457.621149  10.42.3.203 -> 10.63.3.39   TCP 60 http > 55734 [RST, ACK] Seq=1 Ack=1 Win=0 Len=0
458.159020   10.63.3.39 -> 10.42.3.203  TCP 66 [TCP Port numbers reused] 55734 > http [SYN] Seq=0 Win=8192 Len=0 MS
458.159258  10.42.3.203 -> 10.63.3.39   TCP 60 http > 55734 [RST, ACK] Seq=1 Ack=1 Win=0 Len=0
458.689704   10.63.3.39 -> 10.42.3.203  TCP 62 [TCP Port numbers reused] 55734 > http [SYN] Seq=0 Win=8192 Len=0 MS
458.690002  10.42.3.203 -> 10.63.3.39   TCP 60 http > 55734 [RST, ACK] Seq=1 Ack=1 Win=0 Len=0
458.725494   10.63.3.39 -> 10.42.3.203  TCP 66 55736 > http [SYN] Seq=0 Win=8192 Len=0 MSS=1398 WS=4 SACK_PERM=1
458.725696  10.42.3.203 -> 10.63.3.39   TCP 60 http > 55736 [RST, ACK] Seq=1 Ack=1 Win=0 Len=0
459.260930   10.63.3.39 -> 10.42.3.203  TCP 66 [TCP Port numbers reused] 55736 > http [SYN] Seq=0 Win=8192 Len=0 MS
459.261180  10.42.3.203 -> 10.63.3.39   TCP 60 http > 55736 [RST, ACK] Seq=1 Ack=1 Win=0 Len=0
459.795362   10.63.3.39 -> 10.42.3.203  TCP 62 [TCP Port numbers reused] 55736 > http [SYN] Seq=0 Win=8192 Len=0 MS
459.795640  10.42.3.203 -> 10.63.3.39   TCP 60 http > 55736 [RST, ACK] Seq=1 Ack=1 Win=0 Len=0

It just seems to ignore the packets until the NBNS query to which it responds, and then it alternately ignores or RSTs any other packets.

It also does this fun thing where ping works one way, but not the other:

 29.683073  10.42.3.203 -> 10.63.3.39   ICMP 74 Echo (ping) request  id=0x0001, seq=36/9216, ttl=128
 29.688421   10.63.3.39 -> 10.42.3.203  ICMP 74 Echo (ping) reply    id=0x0001, seq=36/9216, ttl=128
 30.758418  10.42.3.203 -> 10.63.3.39   ICMP 74 Echo (ping) request  id=0x0001, seq=37/9472, ttl=128
 30.764715   10.63.3.39 -> 10.42.3.203  ICMP 74 Echo (ping) reply    id=0x0001, seq=37/9472, ttl=128
 31.759546  10.42.3.203 -> 10.63.3.39   ICMP 74 Echo (ping) request  id=0x0001, seq=38/9728, ttl=128
 31.764583   10.63.3.39 -> 10.42.3.203  ICMP 74 Echo (ping) reply    id=0x0001, seq=38/9728, ttl=128
 32.760653  10.42.3.203 -> 10.63.3.39   ICMP 74 Echo (ping) request  id=0x0001, seq=39/9984, ttl=128
 32.766173   10.63.3.39 -> 10.42.3.203  ICMP 74 Echo (ping) reply    id=0x0001, seq=39/9984, ttl=128
 45.221105   10.63.3.39 -> 10.42.3.203  ICMP 74 Echo (ping) request  id=0x0001, seq=4217/30992, ttl=128
 49.749227   10.63.3.39 -> 10.42.3.203  ICMP 74 Echo (ping) request  id=0x0001, seq=4218/31248, ttl=128
 54.747578   10.63.3.39 -> 10.42.3.203  ICMP 74 Echo (ping) request  id=0x0001, seq=4219/31504, ttl=128
 59.754256   10.63.3.39 -> 10.42.3.203  ICMP 74 Echo (ping) request  id=0x0001, seq=4220/31760, ttl=128

Windows firewall is not enabled on the machine, and all machines within HQ can access it just fine. There is a Samba server elsewhere in the network that also works just fine from all offices. It's like these Windows machines are simply refusing traffic from things that are not on their subnet.

Disclaimer: I did not choose either the subnetting or to run a fileshare on a Windows 7 box. They are both before my time and I can't change it at this time. I know they are dumb/bad respectively, please try to look past them. Thanks.

So my coworker set us up a fancy new caching proxy system and then promptly went on vacation. Now I'm getting complaints from our developers/designers that many static resources are being cached for far longer than any of the configuration I've seen dictates.

For example a certain logo file has been changed as of the 13th, but the version from the 9th is still being returned, despite the setting: proxy_cache_valid 200 1h; which should only cache it for 1 hour.

As far as I can see the upstream server is giving Nginx the header Expires: Sat, 14 Feb 2015 19:33:58 GMT and the cache expiry is just running with that regardless of the fact that the Last-Modified: header has changed. I've had a peek at the upstream server's logs and the proxy does not make any attempt to check the status of the file.

How can I get Nginx to check for updated content?

The response headers from the cache:

# curl -v -XHEAD 'http://foo.company.com/inc/skins/pt-1r/schemes/default/img/logo.png'
* About to connect() to foo.company.com port 80 (#0)
*   Trying 1.2.3.4... connected
* Connected to foo.company.com (1.2.3.4) port 80 (#0)
> HEAD /inc/skins/pt-1r/schemes/default/img/logo.png HTTP/1.1
> User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.16.1 Basic ECC zlib/1.2.3 libidn/1.18 libssh2/1.4.2
> Host: foo.company.com
> Accept: */*
>
< HTTP/1.1 200 OK
< Server: nginx/1.0.15
< Date: Thu, 15 Jan 2015 19:37:19 GMT
< Content-Type: image/png
< Connection: keep-alive
< Last-Modified: Fri, 09 Jan 2015 00:04:42 GMT
< Content-Length: 19198
< Cache-Control: max-age=2592000, public, must-revalidate, proxy-revalidate
< Expires: Thu, 12 Feb 2015 22:54:22 GMT
< Vary: User-Agent
< Content-Language: en
< X-Cache-Status: HIT
< Accept-Ranges: bytes

As opposed to directly from the server:

# curl -v --header "Host: foo.company.com" -XHEAD http://10.1.2.3/inc/skins/pt-1r/schemes/default/img/logo.png
* About to connect() to 10.1.2.3 port 80 (#0)
*   Trying 10.1.2.3... connected
* Connected to 10.1.2.3 (10.1.2.3) port 80 (#0)
> HEAD /inc/skins/pt-1r/schemes/default/img/logo.png HTTP/1.1
> User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.16.1 Basic ECC zlib/1.2.3 libidn/1.18 libssh2/1.4.2
> Accept: */*
> Host: foo.company.com
>
< HTTP/1.1 200 OK
< Date: Thu, 15 Jan 2015 19:33:58 GMT
< Server: Apache
< Last-Modified: Tue, 13 Jan 2015 23:04:44 GMT
< Accept-Ranges: bytes
< Content-Length: 45255
< Cache-Control: max-age=2592000, public, must-revalidate, proxy-revalidate
< Expires: Sat, 14 Feb 2015 19:33:58 GMT
< Vary: User-Agent
< Content-Type: image/png
< Content-Language: en

proxy.conf

# Store cached date here
proxy_cache_path /var/lib/nginx/cache levels=1:2 keys_zone=cache:128m inactive=1d max_size=1g;

# Use cache defined above
proxy_cache             cache;
proxy_cache_key         $scheme$host$request_uri;

# Only cache positive responses
proxy_cache_valid       200 1h;
proxy_cache_valid       301 302 5m;

# Temp path for when buffers overflow
proxy_temp_path /var/lib/nginx/temp;

# Buffer data (must be on to allow caching)
proxy_buffering    on;
proxy_buffer_size  128k;
proxy_buffers 100  128k;

# Set some headers
proxy_set_header        Host            $host;
proxy_set_header        X-Real-IP       $remote_addr;

# Die if backend takes too long to connect
proxy_connect_timeout   5;

# Allow adding abcnocache=1 to URLs to skip the cache
proxy_cache_bypass              $arg_abcnocache;

# Add a header showing the cache status
add_header X-Cache-Status $upstream_cache_status;

the site's config:

server {

    server_name foo.company.com *.foo.company.com ;
    listen 80;

    access_log    /var/log/nginx/foo.company.com-access.log;
    error_log     /var/log/nginx/foo.company.com-error.log;

    location / {
        proxy_pass  http://10.1.2.3;
        proxy_redirect default;
    }

}

nginx.conf for good measure:

user nginx;
worker_processes  1;

error_log  /var/log/nginx/error.log;
pid        /var/run/nginx.pid;

events {
  worker_connections  1024;
}

http {

  include       /etc/nginx/mime.types;
  default_type  application/octet-stream;

  access_log    /var/log/nginx/access.log;

  sendfile on;
  tcp_nopush on;
  tcp_nodelay on;

  keepalive_timeout  65;

  gzip  on;
  gzip_http_version 1.0;
  gzip_comp_level 2;
  gzip_proxied any;
  gzip_vary off;
  gzip_types text/plain text/css application/x-javascript text/xml application/xml application/rss+xml application/atom+xml text/javascript application/javascript application/json text/mathml;
  gzip_min_length  1000;
  gzip_disable     "MSIE [1-6]\.";

  server_names_hash_bucket_size 64;
  types_hash_max_size 2048;
  types_hash_bucket_size 64;

  include /etc/nginx/conf.d/*.conf;
  include /etc/nginx/sites-enabled/*;
}

I've got an OpenStack cluster running with about 200 CentOS instances and I've gotten a complaint from my hosting provider that the DNS traffic is getting to be too much for their servers. To address this I've set up a couple bind resolver instances and would like to push these out via DHCP, but I haven't been able to find a way to ensure that the resolv.conf directive options rotate gets either pushed out via dhcp, or can be set via some config file I assume would live in /etc/sysconfig.

I've got about 70 linux instances running on an OpenStack cluster that currently consists of two compute nodes and one controller. Also, these machines live in a RackSpace DC as part of their 'Private Cloud' program, so all of our resources are dedicated.

Previously we were using only RackSpace's NTP servers to synchronize the clocks on all of our instances, but Check_MK was frequently notifying us that the instances were syncing to themselves [stratum 10], implying that the NTP servers were not responding. Given that only 4/70+ instances had public IP addresses I assumed that RackSpace's NTP servers were ratelimiting us since they would be seeing 35+ times the normal rate of NTP queries originating from our two compute hosts. This seemed logical since the 4 instances with public IPs never generated any complaints about NTP.

To address this I changed ntpd.conf on our instances to include our controller node alongside the Rackspace servers so we would at least have a fallback when the RS servers stopped responding. [the NTP cookbook we are using does not allow us to set a preference] However, this has not stopped, or even reduced the number of NTP complaints. I've been seeing last entries in ntpq -p in excess of 60 minutes for all three hosts. I can't see how rate IP-based rate limiting might be coming into effect with the controller node since the instances and the controller reside on, and communicate through, a private network where every instance has its own IP address.

What could be causing this? As far as I've been able to tell there is nothing in the restrict default line that would cause what we're experiencing.

ntp.conf from an instance:

driftfile /var/lib/ntp/ntp.drift
statsdir /var/log/ntpstats/
leapfile /etc/ntp.leapseconds

statistics loopstats peerstats clockstats
filegen loopstats file loopstats type day enable
filegen peerstats file peerstats type day enable
filegen clockstats file clockstats type day enable


server controller01.dfw.domain.com iburst
restrict controller01.dfw.domain.com nomodify notrap noquery
server time.dfw1.rackspace.com iburst
restrict time.dfw1.rackspace.com nomodify notrap noquery
server time2.dfw1.rackspace.com iburst
restrict time2.dfw1.rackspace.com nomodify notrap noquery

restrict default kod notrap nomodify nopeer noquery
restrict 127.0.0.1 nomodify
restrict -6 default kod notrap nomodify nopeer noquery
restrict -6 ::1 nomodify


server  127.127.1.0 # local clock
fudge   127.127.1.0 stratum 10

ntp.conf from the controller node:

driftfile /var/lib/ntp/ntp.drift
statsdir /var/log/ntpstats/
leapfile /etc/ntp.leapseconds

statistics loopstats peerstats clockstats
filegen loopstats file loopstats type day enable
filegen peerstats file peerstats type day enable
filegen clockstats file clockstats type day enable


server 0.pool.ntp.org iburst
restrict 0.pool.ntp.org nomodify notrap noquery
server 1.pool.ntp.org iburst
restrict 1.pool.ntp.org nomodify notrap noquery
server 2.pool.ntp.org iburst
restrict 2.pool.ntp.org nomodify notrap noquery
server 3.pool.ntp.org iburst
restrict 3.pool.ntp.org nomodify notrap noquery

restrict default kod notrap nomodify nopeer noquery
restrict 127.0.0.1 nomodify
restrict -6 default kod notrap nomodify nopeer noquery
restrict -6 ::1 nomodify


server  127.127.1.0 # local clock
fudge   127.127.1.0 stratum 10

• Controller node OS is Ubuntu 12.04.3 LTS running ntpd 4.2.6p3
• Instance OSes are Centos 6.4/6.5 running ntpd 4.2.4p8/4.2.6p5

Edit:

Controller:

# ntpq -npcrv
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
+66.79.167.34    129.6.15.28      2 u  933 1024  377   50.360    3.898   5.064
-208.53.158.34   164.244.221.197  2 u  372 1024  377   27.384    6.635   5.323
+173.230.158.30  199.102.46.73    2 u  780 1024  357   47.656    0.897   0.596
*129.250.35.251  209.51.161.238   2 u  373 1024  377   40.828    1.786   0.163
 127.127.1.0     .LOCL.          10 l  84d   64    0    0.000    0.000   0.000
associd=0 status=0615 leap_none, sync_ntp, 1 event, clock_sync,
version="ntpd [email protected] Tue Jun  5 20:12:08 UTC 2012 (1)",
processor="x86_64", system="Linux/3.2.0-54-generic", leap=00, stratum=3,
precision=-22, rootdelay=48.228, rootdisp=69.214, refid=129.250.35.251,
reftime=d6f049cf.5ce03f06  Wed, Apr  9 2014 22:35:59.362,
clock=d6f04f81.183edd61  Wed, Apr  9 2014 23:00:17.094, peer=21729,
tc=10, mintc=3, offset=1.514, frequency=12.879, sys_jitter=1.158,
clk_jitter=0.896, clk_wander=0.058

Instance:

$ ntpq -npcrv
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
+10.240.0.81     129.250.35.251   3 u 1997 1024  376    0.461   -2.098   0.194
+72.3.128.240    204.9.54.119     2 u 1556 1024  376    0.677    2.234   4.023
*72.3.128.241    204.9.54.119     2 u 1664 1024  376    0.793   -0.783   0.836
 127.127.1.0     .LOCL.          10 l  51h   64    0    0.000    0.000   0.000
associd=0 status=06ff leap_none, sync_ntp, 15 events, stale_leapsecond_values,
version="ntpd [email protected] Sat Nov 23 18:21:48 UTC 2013 (1)",
processor="x86_64", system="Linux/2.6.32-431.5.1.el6.x86_64", leap=00,
stratum=3, precision=-22, rootdelay=30.593, rootdisp=105.114,
refid=72.3.128.241,
reftime=d6f04951.9026bd89  Wed, Apr  9 2014 22:33:53.563,
clock=d6f04fd1.0d15b2be  Wed, Apr  9 2014 23:01:37.051, peer=54008,
tc=10, mintc=3, offset=-0.295, frequency=-0.163, sys_jitter=1.914,
clk_jitter=0.918, clk_wander=0.080, tai=35, leapsec=201207010000,
expire=201306280000

We've recently begun having trouble with web-scraper/DDoS service 80legs taking down our servers a couple times per week due to their abusive crawling practices. Initially we were simply dropping in the following at the bottom of the affected sites' .htaccess files:

<IfModule mod_rewrite.c>
  RewriteEngine On
  RewriteCond %{HTTP_USER_AGENT} ^.*80legs
  RewriteRule .* - [F,L]
</IfModule>

However, it's getting to the point where we just need to block them at the server level across all servers.

According to the Apache docs this config is valid to place in the Server config section, aka httpd.conf, but doing this does not have an effect. Is there a particular approach we can take to block/deny/redirect requests based on User-Agent at the server level on an Apache server with Virtual Hosts enabled?

Note: it is not possible to block this at the firewall level because:

• 80legs uses what is essentially an opt-in botnet to crawl pages. Their last "incident" involved 5250 unique IPs from approximately 900 different networks/IP blocks from around the world.
• We do not currently have the ability to do deep-packet inspection.

We've got a collection of servers that we're backing up using Duplicity. We're trying to build some functionality for our staff so that they can select a file and view the available versions for restore. Duplicity stores its metadata in tar files like:

16M Mar  6 07:20 duplicity-new-signatures.20140305T070733Z.to.20140306T070755Z.sigtar.gz
17M Mar  5 07:17 duplicity-new-signatures.20140304T070728Z.to.20140305T070733Z.sigtar.gz
74M Mar  4 08:02 duplicity-full-signatures.20140304T070728Z.sigtar.gz
13M Mar  3 09:11 duplicity-new-signatures.20140302T070743Z.to.20140303T070723Z.sigtar.gz
14M Mar  2 07:18 duplicity-new-signatures.20140301T070921Z.to.20140302T070743Z.sigtar.gz
18M Mar  1 07:22 duplicity-new-signatures.20140228T071001Z.to.20140301T070921Z.sigtar.gz
16M Feb 28 07:23 duplicity-new-signatures.20140227T071151Z.to.20140228T071001Z.sigtar.gz
15M Feb 27 07:27 duplicity-new-signatures.20140226T070820Z.to.20140227T071151Z.sigtar.gz
13M Feb 26 07:20 duplicity-new-signatures.20140225T071049Z.to.20140226T070820Z.sigtar.gz
14M Feb 25 07:28 duplicity-new-signatures.20140224T070941Z.to.20140225T071049Z.sigtar.gz
92M Feb 24 08:14 duplicity-full-signatures.20140224T070941Z.sigtar.gz

• Each .sigtar.gz is a tar archive containing the signatures of all of the changed files. The signatures are stored as files named identically to those to which they reference.
• duplicity-full files contain a signature for every file in the set
• duplicity-new files contain only signatures for files that have changed since the last full or incremental backup.

Essentially what I need to do is:

for file in `ls /root/.cache/duplicity/hashid/*.sigtar.gz`; do
  tar -tzvf $file signature/path/to/specified/file.name
done

The problem is that even on the server that only contains 1/20th of the amount of data that we expect a 'fully-loaded' server to have, the listing of a 'full' can take in excess of 10 seconds. I shudder to think of how long this process might take once these machines fill up.

Is there any way to speed up the retrieval via tar?

Or if anyone happens to know of a better way to parse Duplicity metadata, I'd definitely like to know.

Short Version: Is it possible to configure spamassassin to perform its own recursive DNS lookups instead of using the server specified by the OS?

Long Version: I've got a collection of dedicated machines at RackSpace running mail services, but I've found that all of my queries to the 'free for most' DNSBL services are failing since they are going through the RS DNS servers rather than being resolved locally. The failure being caused by the squillion other RS customers using the RS DNS servers for the same purpose and pushing them out of the 'free for most' zone and into high orbit.

I am exploring my options and hoping to avoid having to set up my own separate DNS infrastructure.

edit

From what I've heard through other channels is that my two options are:

1. Write a custom patch for SpamAssassin to implement local resolution for DNS/DNSBL queries.
2. Stand up my own DNS server[s].

I've gone the 3/4-assed approach of installing BIND on each machine and specifying 127.0.0.1 as my nameserver. The default configuration [at least in the rpm package] is a resolving-only server that only listens on/allows queries from localhost.

While setting up some hosts in Check_MK for SNMP-only monitoring I've found some hosts where snmpbulkwalk appears to 'hang' and then timeout while processing a certain OID.

eg:

OMD[prod]:~$ snmpbulkwalk -v 2c -c public compute01.domain.com .1.3.6.1.4.1.2021
UCD-SNMP-MIB::memIndex.0 = INTEGER: 0
UCD-SNMP-MIB::memErrorName.0 = STRING: swap
UCD-SNMP-MIB::memTotalSwap.0 = INTEGER: 88109052 kB
UCD-SNMP-MIB::memAvailSwap.0 = INTEGER: 88109052 kB
UCD-SNMP-MIB::memTotalReal.0 = INTEGER: 131860964 kB
UCD-SNMP-MIB::memAvailReal.0 = INTEGER: 94429952 kB
UCD-SNMP-MIB::memTotalFree.0 = INTEGER: 182539004 kB
UCD-SNMP-MIB::memMinimumSwap.0 = INTEGER: 16000 kB
UCD-SNMP-MIB::memShared.0 = INTEGER: 0 kB
UCD-SNMP-MIB::memBuffer.0 = INTEGER: 188772 kB
UCD-SNMP-MIB::memCached.0 = INTEGER: 6685180 kB
UCD-SNMP-MIB::memSwapError.0 = INTEGER: noError(0)
UCD-SNMP-MIB::memSwapErrorMsg.0 = STRING:
UCD-SNMP-MIB::laIndex.1 = INTEGER: 1
UCD-SNMP-MIB::laIndex.2 = INTEGER: 2
UCD-SNMP-MIB::laIndex.3 = INTEGER: 3
UCD-SNMP-MIB::laNames.1 = STRING: Load-1
UCD-SNMP-MIB::laNames.2 = STRING: Load-5
UCD-SNMP-MIB::laNames.3 = STRING: Load-15
UCD-SNMP-MIB::laLoad.1 = STRING: 3.91
Timeout: No Response from compute01.domain.com

snmpwalk, on the other hand, works just fine:

OMD[prod]:~$ snmpwalk -v 2c -c public compute01.domain.com .1.3.6.1.4.1.2021
UCD-SNMP-MIB::memIndex.0 = INTEGER: 0
UCD-SNMP-MIB::memErrorName.0 = STRING: swap
UCD-SNMP-MIB::memTotalSwap.0 = INTEGER: 88109052 kB
UCD-SNMP-MIB::memAvailSwap.0 = INTEGER: 88109052 kB
UCD-SNMP-MIB::memTotalReal.0 = INTEGER: 131860964 kB
UCD-SNMP-MIB::memAvailReal.0 = INTEGER: 94424732 kB
UCD-SNMP-MIB::memTotalFree.0 = INTEGER: 182533784 kB
UCD-SNMP-MIB::memMinimumSwap.0 = INTEGER: 16000 kB
UCD-SNMP-MIB::memShared.0 = INTEGER: 0 kB
UCD-SNMP-MIB::memBuffer.0 = INTEGER: 188772 kB
UCD-SNMP-MIB::memCached.0 = INTEGER: 6685188 kB
UCD-SNMP-MIB::memSwapError.0 = INTEGER: noError(0)
UCD-SNMP-MIB::memSwapErrorMsg.0 = STRING:
UCD-SNMP-MIB::laIndex.1 = INTEGER: 1
UCD-SNMP-MIB::laIndex.2 = INTEGER: 2
UCD-SNMP-MIB::laIndex.3 = INTEGER: 3
UCD-SNMP-MIB::laNames.1 = STRING: Load-1
UCD-SNMP-MIB::laNames.2 = STRING: Load-5
UCD-SNMP-MIB::laNames.3 = STRING: Load-15
UCD-SNMP-MIB::laLoad.1 = STRING: 3.97
UCD-SNMP-MIB::laLoad.2 = STRING: 4.51
UCD-SNMP-MIB::laLoad.3 = STRING: 4.35
UCD-SNMP-MIB::laConfig.1 = STRING: 12.00
UCD-SNMP-MIB::laConfig.2 = STRING: 12.00
UCD-SNMP-MIB::laConfig.3 = STRING: 12.00
UCD-SNMP-MIB::laLoadInt.1 = INTEGER: 397
UCD-SNMP-MIB::laLoadInt.2 = INTEGER: 451
UCD-SNMP-MIB::laLoadInt.3 = INTEGER: 434
UCD-SNMP-MIB::laLoadFloat.1 = Opaque: Float: 3.970000
UCD-SNMP-MIB::laLoadFloat.2 = Opaque: Float: 4.510000
UCD-SNMP-MIB::laLoadFloat.3 = Opaque: Float: 4.350000
...

This is happening across 3 different servers with identical configurations, and I can't find anything in the logs or snmpd config that would seem to indicate any issue.

Any ideas what the issue might be, or what more I can look at?

For some utterly ridiculous administrative reasons we've got a split domain with one mailbox on Office365 which requires us to add include:outlook.com to our SPF record. The problem with this is that that rule alone requires nine DNS lookups of the maximum of 10.

Seriously, it's horrible. Just look at it:

v=spf1
include:spf-a.outlook.com
include:spf-b.outlook.com
ip4:157.55.9.128/25
include:spfa.bigfish.com
include:spfb.bigfish.com
include:spfc.bigfish.com
include:spf-a.hotmail.com
include:_spf-ssg-b.microsoft.com
include:_spf-ssg-c.microsoft.com
~all

Given that we have our own large-ish mail system we need to have rules for a, mx, include:_spf1.mydomain.com, and include:_spf2.mydomain.com which puts us at 13 DNS lookups which causes PERMERRORs with strict SPF validators, and completely unreliable/unpredictable validation with non-strict/badly implemented validators.

Is it possible to somehow eliminate 3 of those include: rules from the bloated outlook.com record, but still cover the servers used by O365?

Edit:

Commentors have mentioned that we should simply use the shorter spf.protection.outlook.com record. While that is news to me, and it is shorter, it's only one record shorter:

spf.protection.outlook.com
  include:spf-a.outlook.com
  include:spf-b.outlook.com
  include:spf-c.outlook.com
  include:spf.messaging.microsoft.com
    include:spfa.frontbridge.com
    include:spfb.frontbridge.com
    include:spfc.frontbridge.com

Edit²

I suppose we can technically pare this down to:

v=spf1 a mx include:_spf1.mydomain.com include:_spf2.mydomain.com include:spf-a.outlook.com include:spf-b.outlook.com include:spf-c.outlook.com include:spfa.frontbridge.com include:spfb.frontbridge.com include:spfc.frontbridge.com ~all

but the potential issues I see with this are:

1. We need to keep abreast of any changes to the parent spf.protection.outlook.com and spf.messaging.microsoft.com records. If anything is changed or [god forbid] added we would have to manually update ours to reflect that.
2. With our actual domain name the record's length is 260 chars, which would require 2 strings for the TXT record, and I honestly don't trust that all of the DNS clients and SPF resolvers out there will properly accept a TXT record longer than 255 bytes.

I have a script that does some housekeeping that works perfectly well when invoked from an interactive shell, but did nothing when invoked by cron. To troubleshoot this I started a shell with a 'blank' environment with the command:

env -i /bin/bash --noprofile --norc

Using this blank env I've dug into my script and found that the following grep will not match any files:

grep -il "^ws_status\s*=\s*[\"']remove[\"']$"

However, when run from an interactive shell the command will return the filenames of the matching files.

As a note, the expression is matching lines like: WS_STATUS = "remove"

Through trial-and-error I discovered that adding -P to the options [Perl regex] the command started working normally in the 'blank' shell. However, I have no idea why my login shell appears to be defaulted to grep -P.

• There is only one grep binary, /bin/grep
• There are no aliases defined for grep=pgrep or grep="grep -P"
• There is no env variable GREP_OPTIONS defined.

What's the deal here?

Note: OS is RHEL v5.10, Bash is v3.2.25, grep is v2.5.1

We need to run pt-stalk on a handful of servers to keep an eye on mySQL, and I was sick of manually starting it every time the server rebooted. A little googling turned up an init script for pt-stalk, and it seemed to work just fine. [my slightly modified version included at the bottom of this post]

It was taking too long to figure out how to push the script and config out via ssh [long story, please don't ask] so I decided to just log into the 20-odd servers and set everything up manually and everything worked.

A couple days later my coworker commented that he was getting the emails, but I clearly wasn't, and it looked like I had put the wrong email in the config. This time I had figured out how to push the change via ssh, and finished everything off with:

for server in `cat serverlist.txt`; do
  ssh -t $server sudo -i service pt-stalk restart
done

And this is the point where pt-stalk stopped working on every single server with:

2013_08_23_11_43_20 Caught signal, exiting
2013_08_23_11_43_20 Exiting because OKTORUN is false
2013_08_23_11_43_20 /usr/bin/pt-stalk exit status 1
2013_08_23_11_43_22 Starting /usr/bin/pt-stalk --function=status --variable=Threads_connected --threshold=100 --match= --cycles=5 --interval=1 --iterations= --run-time=30 --sleep=300 --dest=/var/lib/pt-stalk --prefix= [email protected] --log=/var/log/pt-stalk.log --pid=/var/run/pt-stalk.pid
2013_08_23_11_43_22 Caught signal, exiting

Through yesterday's testing I've deciphered that 'Caught signal, exiting' means it's caught a HUP/TERM/KILL. The first one is from service pt-stalk restart, and the second one immediately after the successful start is from when the ssh session closes. wat.jpg

If I simply ssh to the server, enter sudo -i service pt-stalk start or restart I can log out and it continues happily. However, if I just feed a command to ssh like the above loop pt-stalk it catches a signal and exits. Sometimes it catches two signals before it exits.

What the hell is going on?

My /etc/init.d/pt-stalk for reference:

#!/usr/bin/env bash
# chkconfig: 2345 20 80
# description: pt-stalk
### BEGIN INIT INFO
# Provides: pt-stalk
# Required-Start: $network $named $remote_fs $syslog
# Required-Stop: $network $named $remote_fs $syslog
# Should-Start: pt-stalk
# Default-Start: 2 3 4 5
# Default-Stop: 0 1 6
### END INIT INFO

PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
DAEMON="/usr/bin/pt-stalk"
DAEMON_OPTS="--config /etc/pt-stalk.conf"
NAME="pt-stalk"
DESC="pt-stalk"
PIDFILE="/var/run/${NAME}.pid"
STALKHOME="/var/lib/pt-stalk"

test -x $DAEMON || exit 1

[ -r /etc/default/pt-stalk ] && . /etc/default/pt-stalk

#. /lib/lsb/init-functions

sig () {
    test -s "$PIDFILE" && kill -$1 `cat $PIDFILE`
}

start() {
  if [[ -z $MYSQL_OPTS ]]; then
HOME=$STALKHOME $DAEMON $DAEMON_OPTS
  else
HOME=$STALKHOME $DAEMON $DAEMON_OPTS -- $MYSQL_OPTS
  fi
return $?
}

stop() {
  if sig TERM; then
    while sig 0 ; do
      echo -n "."
      sleep 1
    done
    return 0
  else
    echo "$DESC is not running."
    return 1
  fi
}

status() {
  if sig 0 ; then
    echo "$DESC (`cat $PIDFILE`) is running."
    return 0
  else
    echo "$DESC is stopped."
    return 1
  fi
}

log_begin_msg() {
        echo $1
}

log_end_msg() {
        if [ $1 -eq 0 ]; then
                echo "Success"
        else
                echo "Failure"
        fi
}

case "$1" in
  start)
   log_begin_msg "Starting $DESC"
   start
   log_end_msg $?
   ;;

  stop)
   log_begin_msg "Stopping $DESC"
   stop
   log_end_msg $?
   ;;
  status)
    status ;;

  restart)
    log_begin_msg "Restarting $DESC"
    stop
    sleep 1
    start
    log_end_msg $?
    ;;

  *)
    echo "Usage: $0 {start|stop|status|}" >&2
    exit 1
    ;;
esac

For the general case let's say that the virtual adapter for eth0 has been removed and replaced with another one. Either due to cloning or, more recently, a rash of MAC address conflicts. [Yes, a rash of them.]

Usually I simply rm /etc/udev/rules.d/70-persistent-net-rules and check that nothing in /etc/sysconfig/network* contains a reference to a MAC address and reboot, but this is Linux and there must be a way to avoid rebooting.

Is there a command I can run to pick up the new NIC?

edit

In addition to @dawud's answer below I ran into an issue on my test VM where it had already booted with the new device in place and had it named eth1 in which case udevadm does not seem to 'release' the device. In this case I did the following:

1. Get the MAC address of the device from VSphere.
2. Create/edit /etc/iftab with the line eth0 mac ##:##:##:##:##:## substituting in the MAC address. ifrename does not seem to like it when this file does not exist/contain the name of the destination interface.
3. ifrename -i eth1 -n eth0
4. service network restart and eth0 started up normally.
5. [Optional] rm /etc/iftab so there's one less place for a MAC to live.

I've got a batch machines to update this week, but I'm not quite confident with our established procedure. It basically runs like so for each machine:

1. Mount a shared directory particular to the OS version/bitness, ie: mount -t cifs //server/share/rhel5.3-64/ /mnt/updates/
2. yum update --downloadonly --downloaddir=/mnt/updates/
3. yum update /mnt/updates/*.rpm

We use the mount to reduce the amount of network bandwidth we use up, but since each machine might have wildly different package sets installed there will be packages included in the 'update' command that are not even present on the system, as well as there being multiple older versions of certain packages.

Is this a problem? Will yum skip/remove any unnecessary/obsolete packages before applying the changes?

edit

After reading @aaron-copley's response I decided to do a bit of testing. I logged onto the server, mounted the share, ran yum update --downloadonly --downloaddir=/mnt/updates/, unmounted the share, did a yum clean all, remounted, and re-ran the command. Nothing downloaded. [yay]

I deleted an rpm, ran the command again, and only that one package downloaded. [also yay]

I mounted the share on another box running the same RHEL version, ran yum update --downloadonly --downloaddir=/mnt/updates/, and even though it's marked 221 packages for download, it's only downloading the 30 that were not already in the share. [super yay]

As a bonus, yum also lists the packages that are already downloaded in bold.