Dan Garthwaite's questions

I am giving my developers the gift of kibana using the elasticsearch that is bundled with logstash. They will have kibana in their vagrant boxes.

Goal: Install and configure kibana dashboards via scripts on startup.

Problem: How to pre-load exported dashboards into elasticsearch embedded in logstash?

I have downloaded the dashboards, let's call the file: "fancy.json". Once logstash is up and running how can I automate loading fancy.json into elasticsearch?

I am transitioning servers away from port 22 in an existing multiuser environment. I have configured sshd to listen on two ports: 22 and the new port.

Now I would like to detect when a user connects or is connected to port 22. This is apparently harder than I anticipated.

I tried cranking up the logging in sshd_config but even DEBUG doesn't record the port number.

I am currently scanning netstat's output for TCP connections to port 22 but that lists a hell of a lot of false positives from random bot scanners. [the reason for the port move]

I've seen this question on the mailing list a few times but haven't had a satisfactory answer.

How best to monitor that the pipeline isn't stuck? Clients -> logstash -> elasticsearch.

Logstash and especially elasticsearch are prone to resource starvation. They are both fantastic at picking up where they left off but how, exactly, are people watching their watchers?

Opinions welcome.

There are a number of pkgs that weren't removed with "apt-get purge" and I would like to automate cleaning them up.

You can't apt-get purge apache2.2-common because the pkg is already removed.

You can still list the files that are in the pkg with dpkg -L apache2.2-common.

This means I could just remove the list of files in the pkg, but how would dpkg know this? How do I remove the config files left behind AND remove it from the following report?

Example output:

$ dpkg -l | grep ^rc
rc  apache2.2-common                  2.2.14-5ubuntu8.10                Apache HTTP Server common files
rc  libapache2-mod-php5filter         5.3.10-1ubuntu2ppa6~lucid         server-side, HTML-embedded scripting languag
rc  libapr1                           1.3.8-1ubuntu0.3                  The Apache Portable Runtime Library
rc  libaprutil1                       1.3.9+dfsg-3ubuntu0.10.04.1       The Apache Portable Runtime Utility Library
rc  libgd2-xpm                        2.0.36~rc1~dfsg-3.1ubuntu1        GD Graphics Library version 2
rc  libt1-5                           5.1.2-3ubuntu0.10.04.2            Type 1 font rasterizer library - runtime
rc  php5-gd                           5.3.10-1ubuntu2ppa6~lucid         GD module for php5
rc  ssl-cert                          1.0.23ubuntu2                     simple debconf wrapper for OpenSSL

Thank you, serverfault, for giving me the answer to how-does-an-administrator-generalize-alerting-when-an-event-doesnt-happen.

Now, I only want to check the freshness of these passive services Monday through Friday.

There are some subtleties:

• The status should be allowed to improve over the weekend.
• The status should never degrade over the weekend.
• Notifications should still be emitted over the weekend.

Best I can come up with is YACS - Yet Another Cron Script (to shave). But even then, wouldn't the service definitions need to be toggled every week? Is this service definition dynamism often done with Nagios?

Often my users require me to be just as responsible for knowing if an event hasn't happened.

I've always had to build custom and brittle solutions with cron'ed shell scripts and lots of date edge case testing.

Centralized logging ought to allow for a better, more maintainable way to get a grip on what didn't happen in the last N hours. Something like logstash noticing and nagios alerting.

Update

toppledwagon's answer was so incredibly helpful . o O (Light. Bulb.) that I now have a dozen batch jobs being freshness checked. I wanted to do his thorough answer justice and follow up with how I've implemented his ideas.

I configured jenkins to emit syslogs, logstash catches them and sends status updates to nagios via nsca. I also use check_mk to keep everything DRY and organized in nagios.

Logstash filter

:::ruby
filter {
  if [type] == "syslog" {
    grok {
      match => [ "message", '%{SYSLOGBASE} job="%{DATA:job}"(?: repo="%{DATA:repo}")?$',
                 "message", "%{SYSLOGLINE}" ]
      break_on_match => true
    }
    date { match => [ "timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ] }
  }
}

The magic is in that double pair of patterns in grok's match parameter along with break_on_match => true. Logstash will try each pattern in turn until one of them matches.

Logstash output

We use the logstash nagios_nsca output plugin to let icinga know we saw the jenkins job in syslog.

:::ruby
output {
  if [type] == "syslog"
    and [program] == "jenkins"
    and [job] == "Install on Cluster"
    and "_grokparsefailure" not in [tags] {
      nagios_nsca {
        host => "icinga.example.com"
        port => 5667
        send_nsca_config => "/etc/send_nsca.cfg"
        message_format => "%{job} %{repo}"
        nagios_host => "jenkins"
        nagios_service => "deployed %{repo}"
        nagios_status => "2"
      }
   } # if type=syslog, program=jenkins, job="Install on Cluster"
} # output

icinga (nagios)

Finally, we have arrived at icinga (nagios) by way of nsca. Now we will need passive service checks defined for each and every job we want to notice didn't happen on time. That can be a lot of jobs so lets use check_mk to transform python lists of jobs into nagios object definitions.

check_mk is cool like that.

/etc/check_mk/conf.d/freshness.mk

# check_mk requires local variables be prefixed with '_'

_dailies = [ 'newyork' ]
_day_stale = 86400 * 1.5

_weeklies = [ 'atlanta', 'denver', ]
_week_stale = 86400 * 8

_monthlies = [ 'stlouis' ]
_month_stale = 86400 * 32

_service_opts = [
    ("active_checks_enabled", "0"),
    ("passive_checks_enabled", "1"),
    ("check_freshness", "1"),
    ("notification_period", "workhours"),
    ("contacts", "root"),
    ("check_period", "workhours"),
]

# Define a new command 'check-periodically' that sets the service to UKNOWN.
# This is called after _week_stale seconds have passed since the service last checked in.

extra_nagios_conf += """
  define command {
    command_name check-periodicaly
    command_line $USER1$/check_dummy 3 $ARG1$
  }

  """
# Loop through all passive checks and assign the new check-period command to them.

for _repo in _dailies + _weeklies + _monthlies:
    _service_name = 'deployed %s' % _repo
    legacy_checks += [(('check-periodicaly', _service_name, False), ['lead'])]


# Look before you leap - python needs the list defined before appending to it.
# We can't assume it already exists because it may be defined earlier.

if "freshness_threshold" not in extra_service_conf:
    extra_service_conf["freshness_threshold"] = []

# Some check_mk wizardry to set when the check has passed its expiration date.
# Results in (659200, ALL_HOSTS, [ 'atlanta', 'denver' ]) for weeklies, etc.

extra_service_conf["freshness_threshold"] += [
    (_day_stale,   ALL_HOSTS, ["deployed %s"   % _x for _x in _dailies]  ),
    (_week_stale,  ALL_HOSTS, ["deployed %s"  % _x for _x in _weeklies] ),
    (_month_stale, ALL_HOSTS, ["deployed %s" % _x for _x in _monthlies] ),
]

# Now we assign all the other nagios directives listed in _service_opts

for _k,_v in _service_opts:
    if _k not in extra_service_conf:
        extra_service_conf[_k] =  []

    extra_service_conf[_k] += [(_v, ALL_HOSTS, ["deployed "]) ]