Home / server / Questions / 880336
Accepted
Hakan Baba
Asked: 2017-10-26 15:13:46 +0800 CST

How to add command retry to sensu checks?

  • 772

I have the following sensu check defined in puppet 

sensu::check { 'check_kubernetes_kubelet':
   interval    => 60,
   timeout     => 30,
   command     => 'check_http -H :::name::: -p 10248 -u /healthz -e "200 OK"',
   subscribers => ['kubernetes_minion'],
   contacts    => ['sensu_high'],
}

I want to add command retry to this check. This check should start alerting to its contacts only after N consecutive command executions return error.

In this example, say N is 2. Say a failure happens right after check_X. Then 60 seconds later check_X+1 will return error. After another 60 seconds check_X+2 will also fail. I want the alerting to start after check_X+2's error.

I think sensu filters may be useful. It looks like I need to create a filter like here. But then how do I associate that filter with this check? Do I need to define a custom handler too ?

puppet

2 Answers

  1. You have to use handlers. When your check triggers and event, this gets to the handler or handlers you have defined for it (search for "handler" in checks). Then, each handler can have one or more filters associated.

    From the doc:

    When the Sensu server is processing an Event, it will check for the definition of a handler (or handlers). Prior to executing each Handler, the Sensu server will first apply any configured filter (or filters) for the Handler

    • 1
  2. Best Answer

    Occurences is what you need to use.
    occurrences => 2,

    • 0