RabbitMQ Clustering

By | 03/04/2017

In last post of this series we learnt how to install/configure RMQ for vCloud Director. This post is extension of my last post where I will be adding one more node to my RMQ setup to form a cluster for high availability.

What data is replicated in a RMQ Cluster?

All data/state required for the operation of a RabbitMQ broker is replicated across all nodes. An exception to this are message queues, which by default reside on one node, though they are visible and reachable from all nodes. To replicate queues across nodes in a cluster, see the documentation on high availability

Note: Before proceeding with cluster formation, please ensure following:

1: Use same version of erlang and RMQ server rpm which is installed on master node.

2: RMQ nodes address each other using domain names, either short or FQDNs. Therefore hostnames of all cluster members must be resolvable from all cluster nodes, as well as machines on which command line tools such as rabbitmqctl might be used.

High level steps for cluster formation can be penned as :

  • Have a single node running (rmqsrv01).
  • Stop another node (rmqsrv02).
  • Reset the stopped node (rabbit-2rmqsrv02).
  • Cluster the other node to the root node.
  • Start the stopped node.

Please follow below steps for cluster formation

Step 1) Install same version of erlang and rabbitmq-server rpm which is installed on master node. Steps of doing so have been documented in my previous post

Step 2) Copy Erlang cookie from master node to node 2

RabbitMQ nodes and CLI tools (e.g. rabbitmqctl) use a cookie to determine whether they are allowed to communicate with each other. For two nodes to be able to communicate they must have the same shared secret called the Erlang cookie. The cookie is just a string of alphanumeric characters. Every cluster node must have the same cookie.

On Linux systems, the cookie will be typically located in /var/lib/rabbitmq/.erlang.cookie or $HOME/.erlang.cookie.

scp .erlang cookie file to directory  /var/lib/rabbitmq on 2nd node.

Additionally you can verify the md5sum of the cookie:

Step 3) Stop RMQ app and reset the node.

At this point if you check the cluster status, you will see only one node as disc node and also running_nodes will list only one node i.e. node where you are checking the status.

Step 4)  Add node 2 to cluster

[root@rmqsrv02 ~]# rabbitmqctl join_cluster –ram rabbit@rmqsrv01
Clustering node rabbit@rmqsrv02 with rabbit@rmqsrv01 …

Now if you check the cluster status, you will see both nodes are listed as disc nodes.

Note: By default, the cluster stores messages on the disk. You can also choose to store Queues in Memory. You can have a node as a RAM node while attaching it to the cluster:

# rabbitmqctl stop_app

# rabbitmqctl join_cluster –ram rabbit@rmqsrv02

Node type can be changed via supplying a swaitch ‘change_cluster_node_type’ and then selecting either disc or ram as type

# rabbitmqctl change_cluster_node_type disc | ram

It is recommended to have at least one disk node in the cluster so that messages are stored on a persistent disk and can avoid any loss of messages in case of a disaster

Set the HA Policy

The following command will sync all the queues across all nodes:

I hope this post is informational to you. Feel free to share this on social media if it is worth sharing. Be sociable 🙂

Category: VMware

About Alex Hunt

Hi All I am Manish Jha. I am currently working in OVH US as Operations Support Engineer (vCloud Air Operations). I have around 7 Years of IT experience and have exposure on VMware vSphere, vCloud Director,vSphere Replication, vRealize Automation, NSX and RHEL. If you find any post informational to you please press like and share it across social media and leave your comments if you want to discuss further on any post. Disclaimer: All the information on this website is published in good faith and for general information purpose only. I don’t make any warranties about the completeness, reliability and accuracy of this information. Any action you take upon the information you find on this blog is strictly at your own risk. The Views and opinions published on this blog are my own and not the opinions of my employer or any of the vendors of the product discussed.