Redis vs Kafka vs RabbitMQ

When the use of asynchronous communications is being made for microservices, generally a message broker is used. The role of the broker is to ensure that there is reliability and stability in the communication between different microservices. The broker will also ensure that there is managing and monitoring of messages as well as that there is no loss of messages. There are a few message brokers that can be chosen from, who differ in their capability and scale. The three most popular message brokers that can be chosen from are Kafka, RabbitMQ and Redis. 

Microservices Communication: Synchronous and Asynchronous

The general methods used by microservices for communicating with one another are the synchronous methods and the asynchronous method. Synchronous way of communication involves the caller waiting for a response before it can send the next message. The protocol used for this is REST along with HTTP under it. The asynchronous method of communication involves the messages being sent without having to wait for a response. This is suitable for systems that are distributed and require the message broker to look after the management of the messages. 

For determining which method of communication would be suitable, different parameters should be taken into consideration. These include the structure of the microservices, the infrastructure that is already in place, the scale and latency of the dependencies as well as the purpose of the communication itself. The asynchronous method of communication is complicated during the establishment, it requires the addition of many components onto the stack, but it also has advantages that mostly outweigh its cons. 

Asynchronous Communication Advantages

The definition of asynchronous communications says that it is the non-blocking kind. It also provides better support in terms of scaling than synchronous communication. If there is a crash in the microservices, the asynchronous method of communication provides mechanisms that work as techniques for recovery. This means that it is better in terms of handling errors related to crashes. 

When the use of brokers is made instead of REST protocol, the services involved in the communication do not have to know each other. This means that the introduction of a new service is possible even if some service has been running for some time, meaning there is better decoupling of services. When you have asynchronous operations to choose from, you will be increasing your capability to create a central discovery, monitor and balance the load and also have a policy enforcer. This means you will have flexibility, more capabilities and scalability for the code as well as the system. 

Choosing the Right Message Broker

The management of asynchronous communication is generally handled by a message broker. But other methods also exist, such as asyncio, but they are not used more often because of their scarcity and limitedness. When you are making the choice of a broker for the execution of the asynchronous operations, the following things must be kept in consideration:

• The scale of the broker can be determined by the number of messages that are sent every second by the system.
• The persistence of data is its ability to make a recovery of the messages. 
• You should consider the capability of the consumer, that is if the broker has the capability to manage one-to-one as well as one-to-many type consumers. 

Comparing Different Message Brokers

RabbitMQ (AMQP)

The scale of this message broker on the basis of resources and configuration is approximately 50k messages every second. It is transient as well as persistent in the support it provides the messages. It can handle one-to-many as well as one-to-one customers. 

Released in 2007, RabbitMQ is a very commonly used message broker. It is open source and the delivery of messages is done through pub-sub as well as point-to-point method. It implements Advanced Message Queueing Protocols. The design of this message broker can support complex routing logic. 

Some of the managed services also allow the use of Saas, but it is not provided in the native cloud stack. All of the major programming languages are supported by RabbitMQ, and some issues in the performance can be expected if the mode is persistent. 

Kafka

The scale of Kafka is such that it can be sending up to a million messages every second. It has persistence but can only support the one-to-many type of consumer. 

LinkedIn created Kafka in 2011 for handling high throughput as processing with low latency. As a streaming platform, it is distributed. Kafka is a replica of a publish-subscribe service. There is the provision of data persistence and storage of streams of records, rendering it capable of exchanging high-quality messages. 

Kafka is available as a managed SaaS on Amazon Web Services, Azure as well as Confluent. They are also the creators as well as the main contributors in the Kafka project. All major programming languages are supported by Kafka. 

Redis

The scale of Redis can support the sending of up to a million messages each second. It has an in-memory datastore so it has almost no persistence. It supports both one-on-one and one-to-many types of consumers. 

Redic has a difference when compared to other message brokers. Because the core of Redis is in-memory data storage, it can find its use as key-value high performing storage as well as be used as a message broker. Since there is no persistence, Redis dumps its memory into a Disk/DB. This also makes it good for processing data in real-time. 

Redis was not meant to support one-to-many as well as one-to-many types of consumers originally. But since the introduction of pub-sub in Redis 5.0, its capabilities got a boost and one-to-many became real. 

Message Brokers per Use Case

Till now we have discussed the characteristics of Kafka, RabbitMQ as well as Redis. Although the three discussed here are the most commonly used message brokers, they have very different methods of operating. When deciding on the message broker, they must also be chosen on the basis of their use case. 

Short-lived Messages: Redis

The in-memory database of Redis is a good fit when it comes to using cases that have short-lived messages that do not require persistence. The service provided is fast and it also has in-memory capabilities. Redis is the perfect fit when you need short retention of messages as well as low importance on persistence, meaning some loss can be tolerated. When Redis streams were released, one-to-many use cases also became possible, which was a necessity because of the limitations and pub-sub capabilities. 

Large Amounts of Data: Kafka

Since it has a high throughput, Kafka has a distributed queue that is meant for the storage of large amounts of data and for a long period of time. It also makes it good for one-to-many use cases where there is a requirement for persistence. 

Complex Routing: RabbitMQ

This message broker is old and mature, it has numerous capabilities and features that provide support for complex routing. When the required scale of messages is not high, RabbitMQ is capable of supporting complex routing communications. 

Consider Your Software Stack

The final thing to take into consideration is the current stack of the software. If you want the integration process to be easy, you should not maintain or work with different message brokers in one stack. It becomes quite an easy task to work with a message broker that your software stack supports. When using Celery, for queueing of tasks in your system, it will be easier to work with RabbitMQ or Redis instead of opting for Kafka since it will require rewriting.