What Is Event-Driven Microservices Architecture?

Designing Modern Event-Driven Microservices Applications With Kafka And Docker Containers Suitable For All Levels

Software applications are always required to be maintained and improved. There will always be a need to implement new requirements and enhance existing features. Designing the application right is winning half the battle.

At times, existing features are required to be depreciated. Instead of creating a monolithic application that can become a maintenance nightmare, the industry is moving towards the art of building microservices. I will explain how, when and why we should opt to designing applications using the event-driven microservices architecture.

Microservices architecture is gaining popularity and is being employed in nearly all major software projects.

Most real-time software applications are based on event-driven architecture. Therefore understanding how event-driven microservices applications are designed is extremely important. We can build advanced modern applications when we combine Event-Driven Architecture with the Microservices Architecture paradigm, which are the two concepts I will explain in this article.

Event based architecture is chosen when the system needs to be responsive and is required to respond back at real-time.

Article Aim

This article outlines the concepts of Event-driven microservices architecture.

The article aims to explain what event-driven architecture is, what microservices mean along with their benefits and drawbacks. It will also elaborate on how to design event-driven solutions using Kafka and Docker container technologies.

It All Boils Down To One Concept: We are always required to maintain and improve the software. Therefore, we need to make it easier to enhance our software applications.

• I will start by explaining what event-driven architecture means.
• Then I will outline what microservices are.
• I will then combine the two concepts together to elaborate on how we can build superior and modern software applications using the Event-Driven Microservices architecture.
• I aim to explain the concepts from the basics in simple terms.

I will present the concepts in detail and I recommend everyone to bookmark this article for future read too.

1. What does Event-Driven Mean?

The concept is fairly straightforward.

• The event-driven architecture is based on the concept of a stream of data. Each stream of data raises an event.
• Applications perform actions, such as communicating with each other, once an event is triggered.
• An event is triggered by a service or component of a software application. The nature of an event is that it needs to be immutable.
• An event has a state, such as an event ItemCreated can be triggered to notify that an item has been created. An event can also contain information such as the quantity of the item along with the timestamp and so on.
• For instance, if you are designing a trading application, you can have an PriceTick event whenever a stock price ticks. The event can contain the timestamp along with the price within the event.
• The service/application that produces events is known as the producer.
• The event is communicated and routed to the receivers who then perform respective actions on the events.
• The applications that consume the events are known as the consumers.

The event-driven capabilities within the microservices architecture allow businesses to be more productive. The architecture allows applications and components within applications to communicate with each other in real-time.

2. What Does Microservice Mean?

This section will explain what microservice architecture is.

Microservices are self-contained and serve a single functionality (or a group of common functionalities).

• As the name implies, microservices are essentially independent software services that provide specific business functionality in a software application. These services can be maintained, monitored and deployed independently.

Microservices are built on top of services oriented architecture.

• Microservices architecture is used in large and complex organisations where a number of development teams can work independently to deliver business functionality and where applications are required to serve a business domain.
• Services-oriented architecture (SOA) enables applications to communicate with each other on a single machine or when the applications are deployed on multiple machines across the network.

Monolithic Vs Microservices

Consider the example below where a monolithic application has three major features: User Management, Trade and Calculation features, all sharing a common database.

We can create a microservice for each business feature. Each microservice is meant to be loosely coupled with other services.

For the example above, we now have three services (see below) and each service has its own native database.

Note: The services can also share a common database but then we will have a dependency on a shared database that can restrict our release independence therefore I personally recommend creating a separate database per microservice for complete isolation.

The three services can send each other messages via RestAPI or through a messaging router (more on it below). The issue with communicating over HTTP protocol is that the services are required to know about each other and failure of a service can create a chain effect by impacting all services that depend on it.

Now that we are starting from the basics, let’s understand the benefits and drawbacks of microservices.

3. What Are The Key Benefits Of Microservices?

There are many benefits of microservices architecture, including:

• Each service can be hosted and deployed independently of the other. This results in applications becoming more resilient because each feature can be independently tested and deployed.
• Each team can decide on the contracts of the services and is free to choose the tools and programming language that meet its own requirements, as long as the team respected its contracts and Service Level Agreements (SLAs).
• Having said that, the contracts between the services need to be well defined. The contract includes the message format including the keys and data types of the values.
• All services can communicate with each other even when the services live on different machines. This then allows new functionality to be implemented in the services.
• Microservices encourage organisations to follow automated deployment and continuous delivery.
• As each service is hosted on a separate process, if a service becomes the bottleneck and is a resource-hungry component of the application then it can be moved out into another machine or scaled up without impacting other services. We can also introduce caching on a single service without impacting all of the other services.

For instance, we can use a load balancer such as Nginx to launch multiple instances of a particular service on different machines independently of other services:

If you want to learn how to scale service and balance its load for high-traffic use cases then I highly recommend reading this article:

• It also eliminates a single point of failure as a service can go down without impacting all features that the software application offers.
• Code re-usability is further encouraged because a feature is hosted as a service and it allows multiple services to consume the same feature instead of implementing the same code twice.
• A mix of the latest technology stack can be utilised and embellished to meet the requirements. For instance, R or Python data analysis packages can be deployed and hosted separately and C#.Net can be used to implement services. Additionally, NodeJS can be used on the server side, and AngularJs and ReactJs can be employed to implement the user interface. Each business feature can be implemented using a different technology stack by a different team independent of other features.

4. What Are The Drawbacks Of Microservices?

Microservices architecture has its own drawbacks such as:

1. Separate versioning, monitoring, configuration, documentation, build, deployment and release workflow is required to be built for each microservice which can end up the increasing workload for IT operations teams.
2. It is harder to cancel a running task when services are deployed as separate applications. Microservices introduce problems that are associated with distributed computing such as security, transactions, concurrency, etc.
3. This design can impact performance due to network overhead as each call to service is made across the wire. Caching and concurrency are usually introduced to improve performance.
4. Application-level stability tests are required to ensure new features have not broken the functionality of existing services.

Before I explain event-driven architecture, let’s quickly understand the key microservices issue.

5. The Synchronous Microservices Issue

In a traditional batch system, each interaction with a service starts in a synchronous manner. A service takes the request, performs some actions and then sends a request to another service. If we log all of the interactions in a log file, we can read the log file and understand the flow of data.

For instance, consider this scenario:

Consider a scenario:

1. A trade object is sent to the Trade Service via UI.
2. Trade Service checks the User Management Service if the User who initiated the trade has appropriate permissions to create a Trade.
3. Assuming the user has the required access, the Trade Service then calls the Calculation Service to calculate the required metrics which can be a long-running task.

Considering that the communication is synchronous in nature, the Trade Service will have to wait until the Calculation Service responds. If we extend this synchronous mechanism and add 10 more services in this scenario then the process becomes blocking in nature.

Every request is chained and it becomes increasingly difficult to scale these systems. On top, if a service fails to respond then the user will suffer delays. One can build a Rest API service and each service can communicate with the other by sending HTTP requests. However, this simply moves the problem to a protocol instead of solving the actual problem of synchronous communication.

We can make the communication asynchronous by introducing a state of a request and the Trade Service can continuously poll the Calculation Service for the results but if the Calculation Service breaks down whilst serving the calculation then it can potentially take the Trade Service down too.

There are also other issues. such as that we might not want to expose services over REST API. What if one of the services is faster than the other or is more chatty than the other? Slow services can really block the entire chain.

How do we build applications where the microservices are completely decoupled from each other?

This is where the Event-based driven microservices come in.

6. Let’s Combine The Best Of Both: The Event-Driven Architecture Meets The Microservices Architecture

Event-driven microservices architecture is at the heart of scalable and robust real-time architecture. The core is to break the dependencies between the services and introduce an event-driven architecture where the events are routed via a broker or messaging system.

As the name implies, the event-based architecture is usually chosen when the system needs to be responsive or is required to work at real time.

We can solve the above scenario of Trade and Calculation Service by introducing a Message Broker. Everything starts with an event. Think of an event as a notification. An event can contain the data.

We can invert dependencies, and decouple the services. This then ensures the communication is asynchronous and that there is no single point of failure.

Consider the Trade Service has subscribed to a Trade topic.

1. A trade object is published on the Trade topic.
2. Trade Service gets notified by the Message broker about the event and sends a message to the User Management topic. The User management Service gets notified and checks if the user can perform the task. It then publishes a message to the TradeCheck topic.
3. Trade service gets the message from the TradeCheck topic. It can then check the contents of the message. If the user can execute the calculation then it sends a message to the CalculationTopic.
4. Calculation Service, which has subscribed to the CalculationTopic, gets the event and performs the calculation. It then publishes the results to the CalculationResponse topic when the Calculation completes.

The key to note is that the services do not communicate with each other directly. This allows us to easily scale the architecture.

We could have multiple TradeService, CalculationService and User Management Services that can all subscribe to the topics and they can start publishing to the topics without knowing about each other.

To Summarise:

• When a trade is created, the trader service can raise an event. The trade service will enrich the event message with the trade details. This is the contract.
• The calculation service will then get notified, it will pick the event. The calculation service will parse the message out of the event.

The services can fail independently now.

Each service can be containerised too. I highly recommend containerizing your microservices using Docker technology. Docker is a software platform. It enables software developers to develop, ship and run applications within its containers.

If you are new to Docker then I highly recommend reading this article that explains the concept of Docker and Containers in a clear and succinct manner:

The containers isolate the applications from the environments they run in and help the engineers treat and manage the infrastructure environments as software.

One of the benefits is that Event-Driven Microservices architecture lets us use existing microservices and write new microservices as long as they can publish/subscribe to an event broker system.

7. Let’s Look Into Event-Driven Patterns

• We can store the events in an event source. The events can be stored in the order they arrive. This pattern is known as the Event Sourcing Pattern.
• The key benefit of this pattern is that it allows us to replay the events. This could be useful in a system where we want to skip to the step that failed and retry it post-fixes
• We can also have separate services where one service can subscribe to the read event service and another service subscribes to the write events. This pattern is known as CQRS.

8. Introducing Kafka As A Message Broker Event Source

A potential solution is to introduce a messaging service such as Kafka topics between the microservices. Confluent Kafka can be used as a streaming platform. It can be run on-premises and in the cloud. Kafka is known for its agility and resilience.

• Kafka is one of the most famous event sources.
• Kafka has multiple brokers.
• Each broker can be placed across clusters.
• Each broker has multiple topics.
• Each topic can have multiple consumers and each consumer can consume a message.
• Each message has an offset.
• The messages are stored in a cache and they are also written on disk for failover recoverability scenarios.
• This allows us to replay messages back in the future as long as the messages are not deleted manually or the data retention policy on the Kafka broker permits the data to be stored.

The communication between the service and a topic is done via TCP protocol. TCP is highly performant, backward compatible and a language-agnostic protocol.

• A producer can write anything it wants and is free to choose the schema as long as the consumer can deserialise it.
• There could be 1000s of publishers publishing to a topic and 1000s subscribers reading messages from a topic. The message can be broadcasted to multiple consumers too as shown below:

It doesn’t matter who published the events or who will consume the events, everything will be handled by the Kafka platform.

9. Overcoming The Transaction Related Issues

At times we want synchronous blocking communication and in those scenarios, event-driven architecture is not the architecture to use to design our applications. Hence why it’s vital to understand the business requirements before designing and architecting the application.

One of the hardest problems in an event-driven microservices architecture is to have transactions. It is difficult to start a transaction across multiple services that can be either committed or rolled back upon completion, where the data is required to be idempotent, and where the data is served by multiple microservices asynchronously across the network. One of the common solutions is to use 2PC or Saga design patterns.

• We could use the Two-Phase Commit (2PC) or the SAGA design pattern. In a nutshell, 2PC requires services to commit/roll back a set of actions after the services acknowledge the receipt of the messages. 2PC is useful for short-running intermediate transactions.
• On the other hand, the Saga design pattern allows us to split a transaction into multiple steps. Each mini-transaction serves a functionality and it can be committed or rolled back. The entire transaction is complete and committed when all of the mini transactions within each service has completed.
• We could use Error topics to communicate the rollback. For instance, the Trade service can publish a message to the UserManagement topic once it receives it from the user. The UserManagement topic can pick the message and perform its task. If the User Management service gets an exception then it can publish a message on the Error topic. The Trade service, that has subscribed to the Error Topic, can pick the events and roll back the Trade and it can then notify the user.

Summary

There will always be a need to implement new requirements and enhance existing features.

This article outlined what microservices are along with an overview of their benefits and drawbacks. It then presented how monolithic applications were designed in the recent past and the issues they encountered. Then the microservices architecture was presented along with the technologies which can be used to implement a microservices architecture.

The article then explained what event-driven architecture is, and how it solves the coupling issues related to traditional microservices architecture. It then outlined how Kafka and Docker technologies can be used to design an event-driven microservices architecture where the services are decoupled from each other.

Event-driven architecture can also introduce its own issues such as accomplishing the ACID properties of a transaction. The article explained how to solve those issues and the patterns we can follow.

I hope it helps everyone. Please let me know your feedback or comments in the section below.