Flup's questions

I manage an estate of RHEL workstations all of which use DHCP with static reservations.

On occasion—during a desk move, for example—a workstation might be connected to a port on the wrong VLAN and acquire an address on the wrong subnet. This causes various problems, not least of which is that the Windows-based DHCP and DNS services cause the host's DNS entry to be overwritten with the new (incorrect) address. Users can log in, but can't do anything because hosts on other subnets don't have permission to mount necessary fileservers.

To avoid these problems, I would prefer that incorrectly connected workstations reject DHCP addresses in unexpected subnets. I realise that I could probably blacklist the workstations' MAC addresses in other DHCP scopes but this would be difficult to automate.

Can Linux hosts, in particular RHEL 6 or 7, be configured only to accept DHCP addresses in certain subnets?

I see from this question and answer on Splunk's own Q&A site that it's possible to exclude certain messages from indexing on a Splunk instance.

I have a Splunk Cloud instance where the only way of configuring such things is through the GUI. I don't have access to edit the configuration files directly.

This answer (again on Splunk's site) discusses how to translate entries in props.conf into inputs to the GUI. However my particular case of filtering out messages entirely isn't covered.

How can I do this with the Splunk GUI?

The situation

• Recent upgrade from 2.2 to 3.1(1e).
• Since the upgrade, at 6:51am (UTC+1) every day I experience failures on between zero and three (out of ~60) of the B200-series blades in my installation.
• It's always the same three blades, all in different chassis.
• The failures manifest themselves as a hard hang with 'CPU predictive failure' and 'CATERR_N' messages in the SEL.
• Power-cycling the blade restores it to service (at least until the next failure).
• There are no one-time or recurring schedules in the UCSM that are anywhere near this time of day.
• Cisco TAC is investigating but isn't shedding any light as to why the failures happen at the same time every day.

My research and suspicions

• I have a working theory that these are real hardware problems which have somehow been exposed by the firmware upgrade.
• There's a brief mention of something called the 'sensor scanning manager' in the troubleshooting guide, but I can't find any detail as to what it does or how to monitor it.
• I've all but ruled out an environmental cause. Our power and temperature monitors show nothing unusual at that time. We are not in an earthquake zone :-)

The question

Why are the failures happening at precisely the same time every day?

I use Puppet to generate /etc/exim4.conf and I want to make sure that the configuration is valid before I install the file on production systems.

I've considered—

• using a git hook to call exim4 -bV -C filename... but this won't work because I use an ERB template to generate the file, so the end result isn't actually generated until the Puppet agent runs. I already have a git hook to test ERB syntax.

• letting the init script check the config file... but this isn't good enough because, while the script will refuse to reload Exim if the configuration is invalid, the file will have already been installed and direct calls to Exim (to send mail from applications, for example) will fail.

Ideally what I want is some kind of Puppet directive that looks like

file { '/etc/exim4/exim4.conf':
    content => template("exim/etc/exim4/exim4.conf.erb"),
    notify => Service[exim4],
    but_before_we_install_check_syntax_with => '/usr/bin/exim4 -bV -C',
}

How can I check the syntax of the config file after it's been generated by Puppet but before it gets installed?

I'm using Exim 4.80 and Puppet 2.7.26 on Debian Wheezy systems.

I understand that each mounted ext3 filesystem will have a corresponding kjournald process running. One of those processes is using lots of CPU, and I'd like to know which filesystem is responsible.

I have three ext3 filesystems mounted, but the processes appear only as

root       325     2  0 Sep23 ?        00:30:12 [kjournald]
root      7433     2  0 Sep23 ?        00:00:00 [kjournald]
root      7434     2  0 Sep23 ?        00:09:47 [kjournald]

How can I determine which process is associated with which filesystem?

I'd like to use Nagios to monitor the redundant PSUs in my servers (running Debian Wheezy).

I've run the sensors-detect script in the lm-sensors package, and the only thing it can find is

Driver `ipmisensors':
  * ISA bus, address 0xca2
    Chip `IPMI BMC KCS' (confidence: 8)

I then installed freeipmi-tools, and I find that I can get some useful output from ipmi-sensors:

$ sudo ipmi-sensors --group='Power Supply'
5: Power Supply 1 (Power Supply): [Presence detected]
6: Power Supply 2 (Power Supply): [Presence detected]
7: Power Supplies (Power Supply): [Fully Redundant]

I can write a Nagios plugin to run ipmi-sensors locally, parse its output, and alert if it changes, but I'm reluctant to rely on the output format staying the same, and I can't figure out how to get more machine-readable output.

I've looked at check_ipmi_sensor, but it seems only to operate where the IPMI device is available on the network; mine is not.

Is there a better way than parsing the output of ipmi-sensors?

I have several vhosts served by Apache 2.2.22 on a Debian Wheezy server. Some vhosts are Catalyst apps; all vhosts run with mod_fastcgi and mod_suexec. Each virtual host is configured to use a different ErrorLog.

When code in the Catalyst apps writes to stderr (e.g. warn is called), the message ends up in the main Apache error log instead of the one specified in the <VirtualHost> block. It's not prefixed by a timestamp or any other information.

However, if I call this trivial fastcgi script

#!/usr/bin/perl

use FCGI;

my $request = FCGI::Request();

while($request->Accept() >= 0) {
    print("Content-type: text/html\r\n\r\n");
    warn 'Hello!';
}

the Hello! does end up in the right place, correctly prefixed:

[Thu Jul 10 14:40:18 2014] [error] [client x.x.x.x] FastCGI: server "/data/vhost/wibble/docs/test.cgi" stderr: Hello! at test.cgi line 9., referer: http://my.test.site/

How can I get the Catalyst sites to write their stderr to the correct log?

The only related question I can find is this one -- similar problem but different module (mod_perl), and sadly no answers. I also found this Apache bug report, which describes the same problem with mod_cgid.

Edit: If I put

open STDERR, ">>../../applogs/app.log" or die "can't open applog: $!";

before the Catalyst loop, this redirects stderr to my log file, but obviously no datestamps or other prefixes are added. What I want is for stderr to be treated the same as for a bare FCGI script.

I'm using Varnish 3.0.2 on a Debian Wheezy system. I'd like to be able to set custom headers and have their contents appear in the request logs. However, although I know the headers are being set and passed to the back end, their contents don't seem to be visible to varnishncsa.

In my vcl_recv I have

set req.http.X-my-header = "blahblah";

and, for testing, this is my custom format string for varnishncsa:

'%{X-my-header}i'

I should see blahblah in the log for each request. However, all I get is -. It seems that my header isn't being passed through to varnishncsa.

How can I include my custom headers in my logs?

I'm trying to reference a global hash from a Puppet DSL manifest, but scope.lookupvar is failing to find it.

My declaration is in rubydefs.pp and is of the form

$my_hash = JSON.parse(`...some stuff that spits out JSON...`)

I use this hash in my templates with no trouble.

In my DSL file mymodule.rb I try to reference the hash like this:

foo = scope.lookupvar('::my_hash')
foo.keys.each do |i|
...

but foo always ends up undefined. I've tried just 'my_hash' (without the ::) with the same result. This is the error message from the puppet agent:

err: Could not retrieve catalog from remote server: Error 400 on SERVER: undefined method `keys' for :undefined:Symbol ....

I'm definitely including the files in the right order; my site.pp looks like

import "rubydefs.pp"
...
node mynode {
    include mymodule
}

How can I reference this hash from a Puppet DSL file?

The OpenSSL 'heartbleed' vulnerability (CVE-2014-0160) affects webservers serving HTTPS. Other services also use OpenSSL. Are these services also vulnerable to heartbleed-like data leakage?

I'm thinking in particular of

• sshd
• secure SMTP, IMAP etc -- dovecot, exim & postfix
• VPN servers -- openvpn and friends

all of which, on my systems at least, are linked to the OpenSSL libraries.

Every day I get an email from cron-apt if there are updates available for my Debian systems (I use it in download-only mode). A typical email might look like

CRON-APT RUN [/etc/cron-apt/config]: Tue Aug 27 04:00:03 BST 2013
CRON-APT SLEEP: 866, Tue Aug 27 04:14:29 BST 2013
CRON-APT ACTION: 3-download
CRON-APT LINE: /usr/bin/apt-get dist-upgrade -d -y -o APT::Get::Show-Upgraded=true
Reading package lists...
Building dependency tree...
Reading state information...
The following packages will be upgraded:
   cacti (0.8.7g-1+squeeze1 => 0.8.7g-1+squeeze2)
   [...more updates...]

Usually I'll be happy to apply these updates, but there are occasions when I'll choose not to: perhaps the update fixes a feature which I don't use, but would require a disruptive action (e.g. an Apache restart) to apply.

Is there a way of telling apt 'I'm not interested in that package update, so don't mention it again unless there are further updates'?

I don't want to pin the package, because future updates might be of interest.

I have a server with a big hardware RAID6 array that's been configured with two volumes, presented by Linux as sda and sdb (yeah, I know, but it's a rented colo that came configured this way, and sadly I have to live with it).

I have an I/O-intensive process that uses sdb. I've run with ionice -c3, as I want it to take lower priority than anything else. Unfortunately, ionice doesn't know that the two devices are actually sharing the same array's bandwidth. When that process is busy, the kernel sees that sdb is quiet and gives it all the bandwidth it needs. However, this can make sda very slow, because (as I understand it) ionice works on a per-device basis.

Is there any way of getting the kernel to balance disk I/O across all devices?

The setup

I am collecting statistics from Varnish with Logstash, which is configured to increment statsd counters based on the vhost in the server logs and the result code. I also have carbon creating whisper archives for graphite.

I'm reading logs from varnishncsa which is configured to add vhost and request disposition to the standard logs:

VARNISHNCSA_LOG_FORMAT="%h %l %u %t \"%r\" %s %b \"%{Referer}i\" \"%{User-agent}i\" %{Host}i %{Varnish:hitmiss}x"

My logstash shipper config looks like this:

input {
  file {
    path => "/var/log/varnish/varnishncsa.log"
    type => varnish
  }
}

filter {
  grok {
    type => varnish
    pattern => "%{COMBINEDAPACHELOG} %{NOTSPACE:vhost} %{WORD:varnish_handling}"
    pattern => "%{COMBINEDAPACHELOG}"
  }

  mutate {
    rename => [ 'response', 'status' ]
  }
}

output {
  statsd {
    type => varnish
    host => "my-statsd-host"
    port => 8125
    sender => "%{@fields.vhost}"
    increment => "varnish.response.%{@fields.status}"
    increment => "varnish.handling.%{@fields.varnish_handling}"
  }
}

The problem

Hundreds of distinct counters are being created by carbon due to variations in the domain entered into users' browsers. So, for example, I have

www_mywebsite_com
WWW_MyWebsite_Com
www_mywebsite_net    <-- an alias
...etc...

Obviously these are then missed by my graphs, which only look at statistics under the vhost's canonical name.

What I'd like is for some canonicalising process to happen beforehand. I can write a script to take a 'raw' domain and spit out a 'real' vhost name, but I'm not sure how to integrate that. Do I put it in the logstash config, or in statsd, or carbon? Could I do something with carbon's storage aggregation feature?

Update: I've worked around the worst cases by running carbon's aggregator daemon in front of the cache, and adding rules to rewrite-rules.conf. However, there's very little documentation for that file, and I can't do more powerful things like smash everything down to lowercase.

Apache 2.4 introduced sub-second granularity for log timestamps, so you can do things like

LogFormat "%{%d/%b/%Y %T}t.%{msec_frac}t %{%z}t [...]

As far as I can tell, this feature is not available natively in Apache 2.2. We're not ready to upgrade to 2.4 yet, but I'd still like to have granular timestamps.

Is there a way of getting this functionality in Apache 2.2?

I've just started playing with logstash, and I'm having a problem getting a log shipper process to send events to a redis server on another host.

The output stanza of my logstash config file reads

output {
  redis { host => "11.22.33.44" data_type => "list" key => "logstash" }
}

However, when I run the logstash process, I get error messages containing

:exception=>#<SocketError: Network is unreachable>

A bit of straceing reveals that it's failing here:

connect(13, {sa_family=AF_INET6, sin6_port=htons(6379), inet_pton(AF_INET6, "::ffff:11.22.33.44", &sin6_addr) = -1 ENETUNREACH (Network is unreachable)

i.e. it's trying to connect to the redis host on an ipv6-mapped ipv4 address. The host doesn't have a native ipv6 address and the connection fails.

How can I either get the process to use AF_INET instead of AF_INET6, or somehow configure my host to work around the problem?

In case it's relevant, my java version is

java version "1.6.0_18"
OpenJDK Runtime Environment (IcedTea6 1.8.13) (6b18-1.8.13-0+squeeze2)
OpenJDK 64-Bit Server VM (build 14.0-b16, mixed mode)

I am about to make two-factor authentication mandatory for ssh logins using libpam_google_authenticator. I'd like to be able to waive this requirement when users log in from certain IP addresses.

I have this in /etc/pam.d/sshd:

auth       required     pam_google_authenticator.so

and this in /etc/ssh/sshd_config:

ChallengeResponseAuthentication yes

I thought about using the Match directive in sshd_config, but the manpage suggests that ChallengeResponseAuthentication can't be used in that way. In any event, I'm not sure that would get around the pam requirement.

Is there a way of waiving the challenge-response for clients with certain IP addresses?