Home / server / Questions / 785701
In Process
cdpnet
Asked: 2016-06-23 17:21:23 +0800 CST

How to setup failover with rabbitmq cluster with a loadbalancer like f5

  • 772

I have been trying to find information on f5 developer central and internet regarding the following setup with no luck.

We want to have a rabbitmq cluster with 3 nodes. 1 node will always be primary/master node for ALL queues. 1. Send all connections / traffic for all queues to the current primary node (A). 2. When node A is unresponsive (due to application layer or network layer issues), load balancer should automatically failover all traffic to node (B). 3. If node B fails, move to node C.

The question: How to decide a node is unresponsive and failover to separate node should happen? Is there a way to invoke a call to rabbitmq using amqp protocol through loadbalancer for this purpose? Can't find it well documented.

Even if you don't know on how to implement this with F5, feel free to answer it from a different load balancer or code perspective.

Addition to the question: I would think whatever this health check will be, it has to be concrete enough that rabbitmq cluster would have failed over the master node to node B already, when the LB switch over happens and there was no false alarm.

Thank you for taking time to read and answer.

networking load-balancing failover rabbitmq load-balance

2 Answers

  1. I've done L4 loadbalancing for clustered RabbitMQ with Stingray load balancers - it works well, and we have done RR without any particular issue.

    In the event that one Rabbit node goes down, TCP connections fail and the load balancer sends traffic to the other node.

    Now this is technically inefficient, as any record send to node A will be sent to node B as well and vice versa internally by Rabbit via Erlang's epmd.

    One very important note is that you must set the load balancer to hold TCP connections open indefinitely. This is a common issue, as rabbit MQ uses long running tcp connections but most load balancers are targeted to HTTP-esque connection parameters. Some software (nginx) has very aggressive TCP cleanup windows and will shut these TCP connections, causing an connection failure to occur even though all machines are happy.

    • 4

  2. I would skip the load balancer if you are only following the master, use keepalived and a status check to see if self is master, if it's master then it will use the vip.

    • 0