RabbitMQ Clustering

By | 03/04/2017

In last post of this series we learnt how to install/configure RMQ for vCloud Director. This post is extension of my last post where I will be adding one more node to my RMQ setup to form a cluster for high availability.

What data is replicated in a RMQ Cluster?

All data/state required for the operation of a RabbitMQ broker is replicated across all nodes. An exception to this are message queues, which by default reside on one node, though they are visible and reachable from all nodes. To replicate queues across nodes in a cluster, see the documentation on high availability

Note: Before proceeding with cluster formation, please ensure following:

1: Use same version of erlang and RMQ server rpm which is installed on master node.

2: RMQ nodes address each other using domain names, either short or FQDNs. Therefore hostnames of all cluster members must be resolvable from all cluster nodes, as well as machines on which command line tools such as rabbitmqctl might be used.

High level steps for cluster formation can be penned as :

  • Have a single node running (rmqsrv01).
  • Stop another node (rmqsrv02).
  • Reset the stopped node (rabbit-2rmqsrv02).
  • Cluster the other node to the root node.
  • Start the stopped node.

Please follow below steps for cluster formation

Step 1) Install same version of erlang and rabbitmq-server rpm which is installed on master node. Steps of doing so have been documented in my previous post

Step 2) Copy Erlang cookie from master node to node 2

RabbitMQ nodes and CLI tools (e.g. rabbitmqctl) use a cookie to determine whether they are allowed to communicate with each other. For two nodes to be able to communicate they must have the same shared secret called the Erlang cookie. The cookie is just a string of alphanumeric characters. Every cluster node must have the same cookie.

On Linux systems, the cookie will be typically located in /var/lib/rabbitmq/.erlang.cookie or $HOME/.erlang.cookie.

scp .erlang cookie file to directory  /var/lib/rabbitmq on 2nd node.

Additionally you can verify the md5sum of the cookie:

Step 3) Stop RMQ app and reset the node.

At this point if you check the cluster status, you will see only one node as disc node and also running_nodes will list only one node i.e. node where you are checking the status.

Step 4)  Add node 2 to cluster

[root@rmqsrv02 ~]# rabbitmqctl join_cluster –ram rabbit@rmqsrv01
Clustering node rabbit@rmqsrv02 with rabbit@rmqsrv01 …

Now if you check the cluster status, you will see both nodes are listed as disc nodes.

Note: By default, the cluster stores messages on the disk. You can also choose to store Queues in Memory. You can have a node as a RAM node while attaching it to the cluster:

# rabbitmqctl stop_app

# rabbitmqctl join_cluster –ram rabbit@rmqsrv02

Node type can be changed via supplying a swaitch ‘change_cluster_node_type’ and then selecting either disc or ram as type

# rabbitmqctl change_cluster_node_type disc | ram

It is recommended to have at least one disk node in the cluster so that messages are stored on a persistent disk and can avoid any loss of messages in case of a disaster

Set the HA Policy

The following command will sync all the queues across all nodes:

I hope this post is informational to you. Feel free to share this on social media if it is worth sharing. Be sociable 🙂