Message Queue

Basics, its types and use cases

What is it?

A message queue is an asynchronous service-to-service communication used in serverless and microservices architectures.

Putting the same thing in simple words, it is a process that puts a message onto a message queue and does not require an immediate response to continuing processing

• Message queuing allows applications to communicate by sending messages to each other

• The message queue provides temporary message storage, the purpose of it can be:

  • queue the tasks when the processor is busy or not connected

  • queue the tasks so that they can be processed in parallel (and/or distributed) - (example celery)

  • on-the-fly and real-time processing of data as it arrives (example Kafka as message queue)

Why do we need it?

Instead of building one large application, it is beneficial to decouple different parts of the application and communicate between them asynchronously using message queues

Microservice architecture is very much in use now, few benefits of decomposing an application into different smaller services are:

• Modularity: This makes the application easier to understand, develop, test, and become more resilient to architecture erosion

• Scalability: Since microservices are implemented and deployed independently of each other, i.e. they run within independent processes, they can be monitored and scaled independently.

• Integration of heterogeneous and legacy systems: The process for Software modernization of legacy applications is done using an incremental approach through microservice architecture

• Distributed development: it parallelizes development by enabling small autonomous teams to develop, deploy and scale their respective services independently. Microservice-based architectures facilitate continuous integration, continuous delivery and deployment

Types?

Different types of message queues are in place for different purposes. I have encountered two types of message queues so far, Kafka is one type and Celery (using Redis, RabbitMQ, or AWS SQS) is another type

We can classify based on three properties:

• Message replay is allowed

• We can prioritize a particular task

• Routing is possible; direct or regular expression-based routing allows messages to reach specific queues without additional code

When to use which service:

• Kafka is used more for streaming that is "on-the-fly and real-time processing of data as it arrives", message replay is possible in Kafka that is message is not removed once it’s consumed. But, Kafka lack the ability to prioritize tasks or routing the incoming messages

• Celery with RabbitMQ, Redis, or AWS SQS does not have the option to replay the messages, once the message is acked, it’s removed from the queue. However, it has ability to prioritize the message queues or route the incoming message

Therefore, two message queues have different use cases, and in many cases, two queues compliment each other to add extra capabilities to the system when used efficiently

References for Further Reading