Why do we need to externalize the configuration in Microservices Architecture?
Hello everyone. In this article, we are going to see what is a configuration and look at the problems in maintaining configuration for a microservices architecture. We will then see about Config Server and Config Client and then looks into the steps on how the communication between them happens to work. Then we will look at different sources the config servers can connect to and end the article by looking at the different types of Config Servers. Let's get started.
What is a Configuration?
Configuration is the process of specifying and loading external values to the software application statically or dynamically. One such example is the database config which we typically provide in the property file and use different values for different environments like Dev, QA, and Production.
Configuration is an important part of any software application and it is very important for us to know what our production application consumes i.e the configuration should be readable and properly maintained through a process called Configuration Management.
Microservice configuration management is the process of tracking changes to microservices and the applications that consume them over time.
In a microservices architecture, we have multiple services running and each of them has multiple instances. Hence we need to configure and manage a lot of them and it is critical that we track and manage the version history of configuration changes.
One of the most challenging issues that developers face in a microservice architecture is deciding how to maintain and manage the configuration of the microservices.
Why do we need to externalize the configuration in Microservices Architecture?
In a monolith architecture, there is a single application server running and we traditionally store the application configuration in the environment or application level. Since all the modules of the application are running on a single server, it was easy to handle the config in a single place.
Microservices architecture is a distributed system architecture where there are a lot of independent services running and it is very challenging to maintain the config with the dynamic growth of the application instances based on demand.
Moreover, each microservice needs different configurations for different environments like Dev, QA, UAT, and Prod
Challenges
- How to enable a service to run in multiple environments without modification?
- How to run a microservice in different environments dev, test, QA, staging, production, etc — without modification or restarting the services
- How to handle the configuration of different environments and wire them to the correct environment on which the microservices are running
Solution
The solution to these problems is externalizing the microservices configuration to an external location to handle it from a single place using a dedicated microservice called Configuration Server. This is exactly the same way we manage the centralized logging and monitoring of the microservices architecture.
What are the config server and config client?
The centralized configuration works using a typical client-server architecture pattern. There is a dedicated microservices application called Config Server which has access to the Configuration store like a git repository.
Then each microservices in the system is a Config Client that gets the location of the Config Server from the Discovery server or Service registry during application startup and invokes REST endpoint on the config server to get the properties to inject them in the code. All these communications happen securely using the credentials or SSH keys.
Steps to load the configuration in the microservices
Having seen about Config Server and Client in the above section, let us look at the steps involved in retrieving the configuration to use in the individual microservice.
- The Config Server registers itself with the discovery server upon start-up and hence it knows about the location of the Config Server. User Microservice registers itself with the Discovery Server as well.
2. On start-up, the User microservice asks for the location of the Config Server from the discovery service.
3. Discovery server provides the location of the Config Server to the Config Client i.e User Microservice.
4. User Microservice makes an API call to the Rest endpoint in the Config Server and requests the properties related to it
5. Now, the Config Server loads the relevant property file for the User Microservice based on its environment from the Git Config repository. The Config Server uses an SSH key to make secure communication with the git repository.
6. After retrieving the values, the Config Server provides the config values to the client.
7. Now that the services are up and running, developers or DevOps might update the config in the config store repo which will not the reflected in the User Microservice.
8. Hence, the developers or DevOps must trigger a refresh endpoint in the User microservice, which in turn repeats the above steps to pull the updated config from the config server.
The important point to note is that the configuration values can be updated without restarting the service which is very critical for the microservices architecture
Different config sources
We discussed using the git repository as a source of storing the microservices configuration. There are a lot more options we can use as a config store.
The following sources can be used to store external configurations for microservices
- File System
- Database
- Redis Cache
- AWS S3
- Secrets Manager
- Custom Config store
The Config Server can be used to communicate with different Config Stores one at a time or all at the same time. Then we can specify the order from which the configuration should be fetched. This gives more flexibility but it kind of defeats the purpose of managing the configuration at a centralized place.
Different types of Config Servers
We have readymade Config Server libraries to use in the microservices architecture as follows.
- Spring cloud config server — spring cloud tools that provide the ability to externalize the configurations in distributed systems.
- Consul — Consul is a tool provided by HashiCorp for service discovery and configuration management
- Central Dogma — Central dogma is an open-source configuration server based on git and zookeeper.
- Kubernetes ConfigMap and Secrets — Kubernetes ConfigMap is much like Consul key/value store and the Pods can consume the configuration from the ConfigMap as environment variables, command line arguments, and configuration files.
While the Spring Cloud Config Server can be used only for Spring Boot projects based on Java, the other 3 can be used for developing microservices in other languages like Python, etc.
Summary
In this article, we looked at what is a configuration and the problems in maintaining configuration for a microservices architecture. We about Config Server and Config Client and then looked into the steps on how the communication between them happens. Then we looked at different sources the config servers can connect to and ended the article by looking at the different types of Config Servers.
Hope you liked this article and thanks for reading it!!!
Please go through my below article on how to have a centralized configuration for microservices using Spring Cloud Config
If you like to get more updates from me,
please follow me on Medium and subscribe to email alert.
If you are considering to buy a medium membership,
please buy through my referral link
