Home / server / Questions / 691755
Accepted
Sart
Asked: 2015-05-14 06:55:01 +0800 CST

logstash alert after 1000 occurences

  • 772

I am trying to make Logstash to alert me only after it receives over 1000 items within 10 minutes. I need alerts in both Hipchat and PagerDuty.

My config seems reasonable, but does not work as expected.

filter {
    if my_filtering_conditional_that_is_100%_correct {
        throttle {
            before_count => 1000
            period => 600
            add_tag => ["PD"]
            key => "string"
        }
        clone {
            add_tag => ["Count"]
        }
    }
    if "Count" in [tags] {
        throttle {
            before_count => 1000
            period => 600
            add_tag => ["HC"]
            key => "string"
        }
    }
}

output {
    if "PD" in [tags] {
         pagerduty {
            event_type => trigger
            incident_key => "logstash/Logstash"
            service_key => Pagerduty_API_key
            workers => 1
            description => "Alert message"
        }
    }
    if "HC" in [tags] {
        hipchat {
            color => "random"
            from => "Logstash"
            format => "Alert message"
            room_id => "Room"
            token => "token"
        }
    }
}
logstash

2 Answers

  1. Best Answer

    You may have better success using the metrics filter.

    filter {
  my_filtering_conditional_that_is_100%_correct {
    metrics {
      meter => [ "events" ]
      flush_interval => 600
      clear_interval => 600
      add_tag => "events"
    }
  }
}

output {
  if "events" in [tags] {
    if [events][count] > 1000 {
      # do things
    }
  }
}
    • 4

  2. I think that your best option would be to use http://riemann.io/. It handle events "flows" and that kind of logic wouldn't be to difficult to represent there.

    The example on the following link creates an alert when there are more that 5 events of a certain type:

    (streams
  (where (<= 0 metric 5)
    (with :state "ok" index)
    (else
      (with :state "warning" index))))

    http://riemann.io/howto.html#set-thresholds

    Greetings,

    • 1