Home / server / Questions / 514262
In Process
Goro
Asked: 2013-06-08 15:31:49 +0800 CST

Celery and RabbitMQ - Tools for monitoring and callout alarms

  • 772

I think we've reached the limit of usability of celery events and celery fllower. Is there a good tool for monitoring status of Celery running under RabbitMQ?

In particular, I would like something to raise an alarm when:

  • My queues are backing up
  • Consumers register or de-register

It would be also good to know:

  • How many tasks executed when - some sort of historical metric logs
  • Average task duration, etc
monitoring

1 Answers

  1. I don't use Celery at all but have used RabbitMQ for a while. Right now I use Diamond with the RabbitMQ collector to send stats to Graphite and then monitor the length of every queue. This will at least tell you when your queues are backing up although it doesn't include the memory or disk thresholds used by RabbitMQ. It isn't guaranteed to tell you when consumers register or deregister but it can tell you how many there are. In my case if there are consumers and the queues aren't horribly backed up then there isn't a big enough issue to worry about it.

    There are stats for the number of publishes/deliver/redeliver/acks that you can use to track the history of throughput.

    I track task duration but from what I've seen that's really just a client side issue and will have to be done in Celery directly for which StatsD works really well. Consumer (de)registration may be possible with some hooks into Celery + StatsD publishing.

    For monitoring Graphite I wrote my own tool as a stop-gap but Umpire is useful and there is a Graphite plugin for Nagios.

    • 2