What Is RabbitMQ And How Do You Manage It With Kubernetes?

Organizations are moving from monolithic architecture (where all the code building the application exists as a single, monolithic entity) to microservices architecture as it simplifies app management, making it easier to build, deploy, update, test and scale each service independently without affecting other parts of the architecture.

When the application is built in a microservice architecture pattern, the idea is to develop a single application consisting of two or more small services (microservices). Each microservice is independent, and the application is the sum of all the microservices.

There are different ways to communicate with microservices, such as:

• Brokers (eg. RabbitMQ)
• Remote Procedure Calls (RPC)
• REST APIs

How Microservices communicate?

We have two ways to communicate between microservices:

• Synchronous: Each service calls the other microservice, directly. But, this results in dependency between the services.
• Asynchronous: Let’s say there are two services, Service A and Service B, and you have some central hub or message broker. Here, the two services will act asynchronously. Service A publishes the message in the message queue, and Service B receives the message from the queue. If somehow, Service B is not working, or down due to connectivity issues, the messages will be stored in the queue and once Service B is up and running it will start receiving messages. Until then, the rest of the application will continue operating as usual.

There are several message brokers available in the market which developers are using in their microservice architecture like RabbitMQ, Apache Kafka etc. In this blog post, we will be discussing RabbitMQ.

What is RabbitMQ?

RabbitMQ is an open source message broker. Message broker is a software that enables communication and information exchange between systems, applications and services. They serve as intermediaries between other applications, allowing senders to issue messages without knowing where the receivers are, whether they are active or not, or how many of them are there. RabbitMQ gives your applications a common platform to send and receive messages. And your messages have a safe place to live until they are processed.

Features of RabbitMQ

• Clustering – A RabbitMQ cluster is a logical grouping of one or several nodes, each sharing users, virtual hosts, queues, exchanges, bindings, runtime parameters and other distributed state.
• High Availability.
• Supports several official client libraries for various languages like Java, PHP, Go, Ruby, Python.
• Provides Web-UI for management and HTTP API for monitoring.
• Supports multiple messaging protocols.

Queues in RabbitMQ – Classic vs Quorum

Classic Queues or mirrored queues work with a single leader queue and one or more mirror queues. All reads and writes go through the leader queue, which replicates all the commands (write, read, ack, nack, etc.) to the mirrors. Once all the live mirrors have the message, the leader will send a confirmation to the publisher. Mirroring parameters are configured using policies. A policy matches one or more queues by name (using a regular expression pattern) and contains a definition (a map of optional arguments) that are added to the properties of the matching queues.

Quorum Queues is a replicated queue to provide high availability and data safety. It uses the algorithm of Raft protocol and helps us achieve higher throughput. Each queue is a replicated queue, it has a leader and multiple followers. Clients (publishers and consumers) always interact with the leader, which then replicates all the commands (write, read, ack, etc.) to the followers.

At the time of writing this post, each newly created queue is of classic type. Consider using quorum queues if you are creating new ones, as classic/mirrored queues might get deprecated in later versions.

Managing RabbitMQ using Kubernetes operators

To manage creation, modification & deletion of queues & various other components, RabbitMQ provides us with two Kubernetes operators namely Cluster Kubernetes Operator and Messaging Topology Kubernetes Operator.

• RabbitMQ Cluster Kubernetes Operator: It automates provisioning, management, and operations of RabbitMQ clusters running on Kubernetes. Thus providing us with a consistent and easy method of deployment. These clusters when deployed using the operator can be used by applications running on Kubernetes or outside of Kubernetes.
• RabbitMQ Messaging Topology Kubernetes Operator: It is used with RabbitMQ clusters deployed via the RabbitMQ Cluster Kubernetes Operator. A Messaging topology is the collection of objects such as exchanges, queues, bindings and policies that provides specific messaging or streaming scenarios.

Why HA mode?

RabbitMQ offers features of High Availability (HA). High availability refers to automated failover from one instance to another in case of disk failure or a limited network outage. The impact of failure on availability should be very minimal or should not be seen. Implementation of RabbiMQ in HA mode is advised so that if any of the nodes fails or gets terminated, there is no loss of information and all the queues are intact. This HA mode setup is only required if you want to have replicated queues. If you are creating quorum queues, then you don’t need to do anything extra for HA.

Let us see how we can set up RabbitMQ using Kubernetes operator, along with a policy for HA.