I'm running a single instance of RabbitMQ on a fairly big server (32GB RAM, duel SSD, Octa Core Intel, etc) and have a large number of applications publishing and consuming from RabbitMQ without troubles.
This morning while deploying another application, it simply wouldn't start up complaining that it can't connect. I reduced the number of consumers in my application and it connected just fine. Looking at my RabbitMQ dashboard, I can see the consumer count pegged at 900; increasing the number of consumers in my application again and the application is throwing all kinds of exceptions not being able to connect to RabbitMQ.
So looking at the consumer count that's pegged at 900 and the behavior I'm seeing, I'm guessing I've hit some kind of consumer limit.
Server is idling at 1% and looking at Erlang Processes available, I'm seeing 1048576 available, so plenty capacity left on the server.
How do I increase this artificial limit of 900 consumers?
You probably hit the maximum number of file descriptors. I suppose you are using Linux and many distributions set this ulimit to 1024 by default. When RabbitMQ reaches this limit, it can't accept more connections.
The RabbitMQ website has an article on which network-related parameters you can/should tune, in particular how to increase the file descriptors ulimit.
Seems that RabbitMQ node simply can't handle more connections, resolved it by clustering RabbitMQ.