Home / server / Questions / 692056
Accepted
Garreth McDaid
Asked: 2015-05-15 07:05:02 +0800 CST

RabbitMQ auto-configuration of cluster

  • 772

The RabbitMQ clustering document

https://www.rabbitmq.com/clustering.html

describes a process whereby you can deploy a configuration file for clustering so that a cluster is automatically created when your rabbitmq nodes boot.eg

[{rabbit,
  [{cluster_nodes, {['rabbit@rabbit1', 'rabbit@rabbit2', 'rabbit@rabbit3'], ram}}]}].

You apply this file on each of the 3 nodes at /etc/rabbitmq/rabbitmq.config, and then start rabbitmq on each and a cluster will apparently form.

This doesn't appear to work too well eg

If you start rabbit2, and rabbit3 hasn't already come up, the service will not start on rabbit2, as it is trying to create a cluster with rabbit3.

Equally, if you only apply the config on rabbit1, and have pre-started rabbit2 and rabbit3, rabbit1 will form a cluster with rabbit2 only, as, according to the documentation (I don't understand why):

RabbitMQ will try to cluster to each node provided, and stop after it can cluster with one of them.

The only way this seems to work is if you apply the config file on all 3 nodes, and have them start at exactly the same time. I am trying to deploy this with Ansible, which creates the nodes sequentially, so that doesn't work.

What am I missing here?

ansible

2 Answers

  1. I also deploy RabbitMQ clusters with Puppet, and you actually don't have to spin up all nodes at the exactly the same time.

    What I usually do, and has so far worked for me is:

    • install RabbitMQ (RPM or DEB)
    • set up hosts file on each node, to conatins entries for all the three. example:

    .

    192.168.1.11    dev-c1n01-rabbitmq.example.com  dev-c1n01-rabbitmq
192.168.1.12    dev-c1n02-rabbitmq.example.com  dev-c1n02-rabbitmq
192.168.1.13    dev-c1n03-rabbitmq.example.com  dev-c1n03-rabbitmq

    nodes we're clustering together (because I don't want to rely on DNS being available) * deploy rabbitmq.config

    .

    [
  {rabbit, [
    {cluster_nodes, {['rabbit@dev-c1n01-rabbitmq', 'rabbit@dev-c1n02-rabbitmq', 'rabbit@dev-c1n03-rabbitmq'], disc}},
    {cluster_partition_handling, pause_minority},
    {disk_free_limit, 2147483648},
    {heartbeat, 0},
    {tcp_listen_options, [binary, {backlog, 1024}, {nodelay, true}, {keepalive, true} ]},
    {vm_memory_high_watermark, 0.6},
    {default_user, <<"admin">>},
    {default_pass, <<"somedefaultpass">>}
  ]},
  {kernel, [

  ]}
,
  {rabbitmq_management, [
    {listener, [
      {port, 15672}
    ]}
  ]}
].
% EOF
    • deploy erlang.cookie

    To create erlang cookie, I usually use http://passwordsgenerator.net/ and set it up to create 20 character string consistent only of uppercase characters. Then, put this string into /var/lib/rabbitmq/.erlang.cookie, like this:

    echo -n 'LCQLSHVOPZFHRUXMMAPF' > /var/lib/rabbitmq/.erlang.cookie
    • start nodes (order doesn't matter as long as they have same erlang.cookie and rabbitmq.config)

    This should work for you. Tested on 3.2, 3.3 and 3.5 versions.

    • 3
  2. Best Answer

    I've made some progress on this.

    The issue appears to relate to the choice of using ram nodes rather than disc nodes in the rabbitmq.config file. From the documentation:

    RAM nodes are an advanced use case; when setting up your first cluster you should simply not use them. You should have enough disc nodes to handle your redundancy requirements, then if necessary add additional RAM nodes for scale.

    A cluster containing only RAM nodes is fragile; if the cluster stops you will not be able to start it again and will lose all data. RabbitMQ will prevent the creation of a RAM-node-only cluster in many situations, but it can't absolutely prevent it.

    When I change the config file to use "disc" instead of "ram", cluster creation was a lot more stable.

    [{rabbit,
    [{cluster_nodes, {['rabbit@rabbit1', 'rabbit@rabbit2', 'rabbit@rabbit3'],disc}}]}].
    • 0