Everything about Distributed Message Queue

With the growing adoption of Distributed System movement across Industry, Message Queue is now becoming part of this significant movement. Data flow from end-users to applications is growing tremendously. With this, we also need to understand the nature of these requests, i.e., whether the request must be synchronously or can it be taken offline and executed offline? If asynchronous, we need a message queue to accommodate multiple such asynchronous nature requests.

What is Message Queue

A message queue is a messaging destination that uses the queue data structure to facilitate asynchronous communication between two services, commonly used in serverless and microservices architectures. Messages are stored on the queue until they are processed and deleted. Each message is processed only once by a single consumer. And MQ helps to decouple heavyweight processing, buffer or batch work, and smooth spiky workloads.

The message queue is comprised of two terms:

Message: this is the object passed from the producer to the Consumer. The object can be requests, information, meta-data, etc.
Queue: this is a temporary buffer that stores messages. It uses the First-In-First-Out method to pass the messages from producer to Consumer.

The producer creates the message and sends it to the message queue if the Consumer is busy processing it immediately, the queue stores it until the Consumer is available.
The Consumer retrieves the message from the queue and starts processing it.
The message queue then temporarily locks the message to prevent it from being read by another consumer.
After the Consumer completes the message processing, it deletes the message from the queue to prevent it from being read by other consumers.

This architecture provides better performance because of Async in nature and removes dependencies on multiple systems at once.

Working With REST API

Instead of creating the actual resources, create a temporary one. Instead of returning a 201 (Created) HTTP response, you can issue a 202 (Accepted) response code. It informs the client that the server has accepted and understood the request, but the resource is not (yet) created. Send the temporary resource inside the Location header. Request:

POST /blogs HTTP/1.1
<xml>
    blogdata
</xml>

Response:

HTTP/1.1 202 Accepted
Location: /queue/12345

Applications of Message Queue

Notification Engine
Schedule Jobs
Priority Tasks
Streaming Service

This location can store information about the status of the actual resource: an ETA. Once the actual resource is available, The temporary resources can return a 303 (See other) response. The location header returns the URI to the definitive resource. A client can DELETE the temporary resource, or the server can expire and return a 410 (Gone) later on.

Message Queue vs. Message Bus

A Message Bus is a messaging infrastructure that allows different systems to communicate through a shared set of interfaces (message bus).

The basic idea of a message queue is a simple one:

Two (or more) processes can exchange information via access to a standard system message queue.
The sending process places a message onto a queue via some (OS) message-passing module that another process can read.

Message Queue contains FIFO(first in, first out) rule, whereas Message Bus does not.

Message Queue vs. Shared Memory

It would be for multiple reasons; let us try to break this into numerous points for simplification −

Shared memory data need to be protected with synchronization when multiple processes communicate at the same time. If the frequency of writing and reading using the shared memory is high, so implementing the functionality would be very complex. Not worth concerning utilization in this kind of case.
If all the processes do not need to access the shared memory but very few only need it, it would be better to implement it with message queues.
If we want to communicate with different data packets, process A sends Object 1 to process B, Object 2 to process C, and Object 3 to process D. In this case, it is simpler to implement with message queues.
Of course, the order of the message queue is FIFO (First In First Out). The first message inserted in the queue is the first one to be retrieved.

Message Queue vs. Message Broker

A message broker (also known as a service bus) is a piece of middleware responsible for persisting and routing messages while allowing you to decouple your system into smaller parts. A message queue is a part of a message broker and is just a persistence mechanism.

The features of Message Queue

Now, let’s discuss the qualities we should be having for Message Queue.

Pull / Push Delivery provides push and pull options for retrieving messages. Pull means continuously querying the queue for new messages. Push means that a consumer is notified when a message is available (also called Pub/Sub messaging). You can also use long-polling to allow pulls to wait a specified amount of time for new messages to arrive before completing.
Schedule / Delay Delivery supports setting a specific delivery time for a message. You can set up a delay queue if you need to have an expected delay for all messages.
At-Least-Once Delivery: may store multiple copies of messages for redundancy and high availability and resend messages in the event of communication failures or errors to ensure they are delivered at least once.
Exactly-Once Delivery: When the system cannot tolerate the duplicates execution, the FIFO message queue will ensure that each message is delivered exactly once (and only once) by filtering out duplicates automatically.
FIFO Queues: In these queues, the oldest (or first) entry, sometimes called the “head” of the queue, is processed first.
Dead-letter Queues: A dead-letter queue is where other queues can send messages that can’t be processed successfully in a regular queue. This approach makes it easy to set them aside for further inspection without blocking the queue processing or spending CPU cycles on a message that might never be consumed successfully.
Ordering: Most message queues provide best-effort ordering, which ensures that messages are generally delivered in the same order as they’re sent and that a message is delivered at least once.
Poison-pill Messages: Poison pills are particular messages that can be received but not processed. They are a mechanism used to signal a consumer to end its work, no longer waiting for new inputs, and are similar to closing a socket in a client/server model.
Security: Message queues will authenticate applications that try to access the queue and allow you to use encryption to encrypt messages over the network as well as in the queue itself
Filtering: This feature empowers the subscriber to create a message filtering policy only to get the notifications it is interested in, instead of receiving every message posted to the topic.
Message Priority: A program can prioritize a message when it puts the message in a queue. This approach determines the position in the queue at which the new message is added. Programs can get messages from a queue either in the FIFO order or by getting a specific message. (A program might want to get a particular message if it is looking to reply to a request that it sent earlier.). You can achieve this with Pub/Sub message queue quickly.
Durability: Message queues temporarily store messages until they are received and processed successfully by the Consumer.
Reliability: Queues make your data persistent and reduce the errors when different parts of your system go offline. By separating various components with message queues, you create more fault tolerance. If one part of the system is ever unreachable, the other can continue interacting with the queue. The queue itself can also be mirrored for even more availability.
Scalability: We can quickly increase the number of workers to process the high number of pending tasks.

Types of Message Queue

Point-to-Point Message Queue

This queue type is sent from one application(producer/ sender) to another application(consumer/receiver) via a queue. There can be more than one consumer listening in a queue, but only one of them will be able to get the message. Hence, it is Point to Point or One to One.

Pub / Sub Message Queue

Publish/subscribe messaging, or pub/sub messaging, is a form of asynchronous service-to-service communication used in serverless and microservices architectures. In a pub/sub model, any message published to a topic is immediately received by all subscribers to the topic. Pub/sub messaging helps to enable event-driven architectures or to decouple applications to increase performance, reliability, and scalability.

The Publish-Subscribe model allows messages to be broadcast to different system parts asynchronously. A sibling to a message queue, a message topic provides a lightweight mechanism to broadcast asynchronous event notifications and endpoints that allow software components to connect to the topic to send and receive those messages. Unlike message queues, which batch messages until they are retrieved, message topics transfer messages with no or very little queuing and push them out immediately to all subscribers. All components that subscribe to the topic will receive every message broadcast unless the subscriber sets a message filtering policy.

The subscribers to the message topic often perform different functions and can do something different with the message in parallel. The publisher doesn’t need to know who is using the information it is broadcasting, and the subscribers don’t need to know who the message comes from. This messaging style is a bit different from message queues, where the component that sends the message often knows the destination it is sending to.

Scaling Consideration

Message queues make it possible to scale precisely where you need to. When workloads peak, multiple instances of your application can all add requests to the queue without risk of collision. As your queues get longer with these incoming requests, you can distribute the workload across a fleet of consumers. Producers, consumers, and the queue can all grow and shrink on demand.

Here are some of the strategies you can use to scale:

Increase Workers if the rate of message production is higher than the rate of consuming
Partitioning: Topics can have many partitions. It helps in parallelism.

Error Handling

Separate Queue like Dead-letter Queue.
Monitor and Identify the unhealthy workers or instances and replace them.

Last Thought

There is no doubt that the message queue is the backbone of a distributed system. You can configure your MQ to suit your need and relax on some NFRs and stricken on some. We can quickly scale and improve performance as compared to live traffics. Many message servers have an in-built setup for replications to provide fault-tolerant and durable data. Here are a few last tips:

Add standard metadata to each event, such as Origin-UUID (pass on when triggering other events) and Seen-By.
The document which service consumes/produces which events.
Only include data relevant to the event; fetch other data as needed.

If you are interested in learning more about internal working on message queues, I recommend checking out Kafka. Many message brokers or managed message queues like SQS, RabbitMQ, ActiveMQ, etc. Before proceeding with any random service, list down the need of your message queues and then pick one with the most suitable to your requirement.

Here, I conclude this learning; I hope you have learned something new today. Please do share with more colleagues or friends. Finally, Consider becoming a Medium member. Thank you!