Summary

The web content provides a comparative analysis of various Message Queueing (MQ) and Event Streaming Providers (ESP), including Apache Kafka, Apache Pulsar, AWS Kinesis, Amazon SQS, RabbitMQ, IBM MQ, RocketMQ, and Azure Queue Storage, highlighting their features, pros, and cons.

Abstract

The article "Message Queueing and Event Streaming Providers in Comparison" offers an overview of the current landscape of messaging and streaming platforms. It discusses the functionalities, advantages, and limitations of eight prominent solutions: Apache Kafka, Apache Pulsar, AWS Kinesis, Amazon SQS, RabbitMQ, IBM MQ, RocketMQ, and Azure Queue Storage. The text underscores the importance of these technologies in modern IT for handling real-time data streams and asynchronous service communication, particularly in microservices and serverless architectures. Each provider is evaluated based on criteria such as open-source status, latency, throughput, scalability, ease of deployment, documentation, community support, and integration capabilities. The article also touches on the historical context of how ESPs have evolved to address the shortcomings of traditional MQs, such as data reprocessing, ordering, and retention.

Opinions

Apache Kafka is recognized for its distributed event storage, low latency, and high throughput but is noted for its complex code base and the need for additional functionality development.
Apache Pulsar is praised for its ease of deployment and scaling, as well as geo-replication, but is considered less mature and less documented compared to Kafka.
Amazon Kinesis is appreciated for its seamless integration within the AWS ecosystem and serverless capabilities but is criticized for its confusing documentation and dependency on Redis.
RabbitMQ offers delivery guarantees and redundancy but is seen as slower for large datasets and has a limited range of brokers.
IBM MQ is acknowledged for its security and ease of use but is critiqued for message prioritization issues and high costs.
Amazon SQS provides fault tolerance and reduces administrative expenses but lacks performance control and has high costs at scale.
RocketMQ is highlighted for its high performance and support for both point-to-point and publish-subscribe models but lacks message tracking capabilities.
Azure Queue Storage is noted for its security, integration with Azure services, and cost-effectiveness but lacks message ordering and subscription mechanisms, and has a size limitation for messages.
The article suggests that ESPs have been introduced to overcome the limitations of Messaging Middlewares, such as repeated data consumption, unordered data processing, and short data retention times.

Message Queueing and Event Streaming Providers in Comparison

Apache Kafka, Apache Pulsar, AWS Kinesis, Amazon SQS, RabbitMQ, IBM MQ, RocketMQ, Azure QS

Event Streaming Platforms (ESP) and Message Queues (MQ) are pretty popular solutions in nowadays IT world. Event Streaming cops with getting real-time data from different sources as a stream of event and delivers this continuous flow of data to different destinations for further refinement. MQ is a form of asynchronous service-to-service communication used in serverless and microservices architectures. Messages are stored on the queue until they are processed and deleted. Each message is processed only once, by a single consumer.

Event Streaming is used widely, for example, in industries like:

continuous capturing of data from different IoT sensors/devices
handling with customer data in e-commerce, hotel and travel industry
processing financial transactions in bank, insurance companies

ESP versus MQ…

Functionalities of Event Streaming Platforms and Message Queues are somehow interchangeable. They both provide decoupled message like communication over network, message production/consumption is asynchronous, the server/client model is highly decoupled. Historically, ESPs are introduced as a predecessor or competitor to MQs while eliminating shortages of Messaging Middlewares like:

repeated consumption of data
ordered consumption of data
long time data retention

So let us have a quick overview of current ESP and MQ providers in market and analyse their pros and cons:

Apache Kafka: distributed event store and stream-processing platform.

+ Open source + Low latency + High throughput + Real-time processing + Scalable + Well documented - Lots of Detail Functionalities should be built on top - difficult to understand code base (mix of Java/Scala)

Apache Pulsar: an all-in-one messaging and streaming platform

+ Open source +Easily deployable + Written compoletely on java + Easy to scale up/down + Geo-replication - Less mature (than Kafka) - Less documented/smaller community (than Kafka)

Amazon Kinesis: performant streaming platform

+ Well integrated into AWS Ecosystem + Scalable + Totally serverless platform + Simple management - Documentation is confusing. - Difficult VPC integration - Dependent on redis - Uses Lots of storage

Rabbit MQ: is an open-source message-broker software

+ Delivery and order guarantee + Redundancy (persisting until they are processed completely) + Decoupling + Scalability - Slow at processing large datasets - Quite tricky at maintaining - It has premium integration services - Limited range of brokers as it is a single broker based - Poor documentation

IBM MQ: is a family of message-oriented middleware products

+ Security + Ease of use - Problems with message prioritization - Difficult to integrate - Costly

Amazon SQS: is a distributed message queuing service

+ Reliably deliver messages + Fault tolerance + Reduce Administrative Expenses + Ensure the safety of sensitive information + Scale elastically and efficiently - Less performance control - Scale-up cost high - Costs are high at scale - Absence of support for broadcasting

RocketMQ: is a distributed messaging and streaming platform

+ High performance + Supporting both point-to-point and publish-subscribe + Low latency - No message tracking - Smaller community

Azure Queue Storage: a service for storing large numbers of messages

+ Not expensive + Secure + Well integrated to Azure Ecosystem + Well documented - Messages cant be ordered - No Subscribtion mechanism - Size Limitation of messages (64KB)

If you are interested in more Software Architecture topics take a look at below list.

Relevant Articles:

P.S. You can connect with me on twitter or linkedin.