I am working on a project which requires rolling out new celery servers if the broker(redis) queue is consistently higher than a predetermined threshold size and killing the new boxes when the queue size comes down. I have scripts to take care of introducing new boxes and making sure right configuration is running on them. I plan to check the queue by redis-cli after frequent(15 mins) intervals and if size is greater then threshold bring down the time gap between check(say 3 mins) check twice and then roll out new boxes and follow the same approach to kill the new servers.
I am new to celery and it would be great if I can get some advise on this approach.
Thanks
If you are using an EC2 autoscaling group, this can be quite simple. I'm doing something similar, but with RabbitMQ queue length.
Check queue length periodically, and then decide for a certain queue length, how many workers you want. You can then use the ASG API (or command line tools) to set the "desired capacity" for the group according to the number of workers you need. Then the ASG takes care of creating and terminating instances as needed.