How does an administrator generalize alerting when an event doesn't happen?

I setup passive checks in nagios for various events. Then at the end of the event the passive check is sent to nagios (either via wrapper script or built into the event itself.) If the passive check hasn't been received in freshness_threshold seconds, it will run check_command locally. check_command is setup as a simple shell script which returns critical and the information of the service description.

I don't have code examples handy, but if I could if interest is shown.

EDIT ONE added code examples:

This assumes that you have done the basic setup for NSCA and send_nsca (make sure password and encryption_method is the same in send_nsca.cfg on the client and nsca.cfg on the nagios server. Then start nsca daemon on the nagios server.)

First we define a template that other passive checks can use. This goes into services.cfg.

define service {
    name                    standard-passive-service-template
    active_checks_enabled   0
    passive_checks_enabled  1
    check_freshness         1
    max_check_attempts      1
    normal_check_interval   10
    retry_check_interval    5
    contact_groups          sysadmins
    notification_interval   0
    notification_options    w,u,c,r
    notification_period     24x7
    check_period            24x7
    check_command           check_failed!$SERVICEDESC$
    register                0
}

This says that if a notification hasn't come in, run check_failed with $SERVICEDESC$ as an argument. Let's define the check_failed command in commands.cfg.

define command {
    command_name     check_failed
    command_line     /usr/lib/nagios/plugins/check_failed $ARG1$
}

Here is the /usr/lib/nagios/plugins/check_failed script.

#!/bin/bash
/bin/echo "No update from $*. Is NSCA running?"
exit 2

Having an exit of 2 makes this service critical according to nagios (see below for all nagios service states.) Sourcing /usr/lib/nagios/plugins/utils.sh is another way, then you could exit $STATE_CRITICAL. But the above works even if you don't have that.

This gives the added notice of "Is NSCA running" because it might be the case that the service didn't check in properly OR it might be the case that NSCA has failed. This is more common than one might think. If multiple passive checks come in at once, check for a problem with NSCA.

Now we need a passive check to accept the results. In this example I have a specially crafted cron job that knows about all of the different types of raid controllers in our environment. When it runs it sends in a notification to this passive check. In this example, I don't want to be woken up in the middle of the night (edit notification_period as needed.)

define service {
    use                     standard-passive-service-template
    hostgroup_name          all
    service_description     raidcheck
    notification_period     daytime
    flap_detection_enabled  1
    freshness_threshold     7500 # 125 minutes
    notification_options    c
    is_volatile             0
    servicegroups           raidcheck
}

Now there's the cronjob that sends info back to the nagios server. Here's the line in /etc/cron.d/raidcheck

0 * * * *  root  /usr/local/bin/raidcheck --cron | /usr/sbin/send_nsca -H nagios -to 1000 >> /dev/null 2>&1

See man send_nsca for options, but the important parts are 'nagios' is the name of my nagios server and the string that is the printed at the end of this script. send_nsca expects a line on stdin of the form (perl here)

print "$hostname\t$check\t$state\t$status_info\n";

$hostname is obvious, $check in this case is 'raidcheck', $state is the nagios service state (0 = OK, 1 = warning, 2 = critical, 3 = unknown, 4 = dependent.) and $status_info is an optional message to send as the status info.

Now we can test the check on the command line of the client:

echo -e "$HOSTNAME\traidcheck\t2\tUh oh, raid degraded (just kidding..)" | /usr/sbin/send_nsca -H nagios

This gives us a nagios passive check that expects to be updated every freshness_threshold seconds. If the check isn't updated, check_command (check_failed in this case) is run. The example above is for a nagios 2.X install, but will likely work (maybe with minor modification) for nagios 3.X.

Not sure which type you are referring to as the "event doesn't happen" can take different forms, it can be either conditional or unconditional. Examples:

• user authentication failures not followed by successful login indicates the user has forgot her password (or bruteforce attempt)
• no user authentication during the day - user didn't show up for work

If you are after the first case and need an open source tool, there is a Pairwithwindow rule in SEC and an Absence rule in nxlog.(Note that I'm affiliated with the latter).

The second type is simpler and both tools can handle that too afaik.