Monitoring Docker Containers with Grafana, Loki, and Promtail
![](https://cdn-images-1.readmedium.com/v2/resize:fit:800/1*AKqw388SAaGp0GENx57haQ.png)
In this blog, I will teach :
- How you can run Grafana, Loki, and Promtail using docker-compose.
- How you can monitor the docker logs using Grafana
Prerequisites:
Make sure your system has the following tools installed
- Docker
- Docker Compose
To install them use the below command:
- Docker:
sudo apt-get update sudo apt install docker.io
- Docker Compose:
# Download Docker Compose Binary
sudo curl -L "https://github.com/docker/compose/releases/latest/download/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
#Make Binary Executable
sudo chmod +x /usr/local/bin/docker-compose
#Add User to Docker Group
sudo usermod -aG docker $USER
#Verify Installation
docker-compose --version
Configurations:
To install the Grafana, Loki, and Promtail using Docker Compose we need two YAML files: docker-compose.yaml
and loki-config.yaml.
docker-compose.yaml:
version: '3'
services:
loki:
image: grafana/loki:latest
ports:
- "3100:3100"
networks:
- loki-net
promtail:
image: grafana/promtail:latest
volumes:
- ./promtail-config.yaml:/etc/promtail/promtail-config.yaml
networks:
- loki-net
depends_on:
- loki
grafana:
image: grafana/grafana:latest
ports:
- "3000:3000"
environment:
- GF_SECURITY_ADMIN_PASSWORD=admin
networks:
- loki-net
depends_on:
- loki
networks:
loki-net:
external: true
Explanation:
Services:
- Loki: Uses the official Grafana Loki Docker image. It exposes port 3100 and connects to the “loki-net” network.
- Promtail: Also uses the official Grafana Promtail Docker image. It mounts a local configuration file
promtail-config.yaml
into the container at/etc/promtail/promtail-config.yaml
. It connects to the "loki-net" network and depends on the Loki service. - Grafana: Uses the official Grafana Docker image, exposes port 3000, sets the admin password to “admin” using an environment variable, and connects to the “loki-net” network. It depends on the Loki service.
Networks:
- loki-net: Declares a custom network named “loki-net” with the
external: true
flag. This suggests that the network is created externally (possibly before running this Docker Compose file) and is not managed by Docker Compose itself.
Version:
- Specifies the version of the Docker Compose file format. In this case, it’s version 3.
loki-config.yaml:
server:
http_listen_port: 9080
grpc_listen_port: 0
positions:
filename: /tmp/positions.yaml
clients:
- url: http://loki:3100/loki/api/v1/push
scrape_configs:
- job_name: system
static_configs:
- targets:
- localhost
labels:
job: varlogs
__path__: /var/log/*log
- job_name: docker
static_configs:
- targets:
- localhost
labels:
job: docker_logs
__path__: /var/lib/docker/containers/*/*-json.log
Explanation:
Server Configuration:
http_listen_port: 9080
: Prometheus will listen for HTTP requests on port 9080.grpc_listen_port: 0
: Disables gRPC (Google Remote Procedure Call) for this Prometheus server.
Positions Configuration:
filename: /tmp/positions.yaml
: Specifies the file path where Prometheus will store the current positions in the logs. This information is used to resume scraping after a restart.
Clients Configuration:
- url: http://loki:3100/loki/api/v1/push
: Configures Prometheus to push logs to Loki, a horizontally scalable, highly available log aggregation system.
Scrape Configurations:
System Job:
job_name: system
: Defines a job named "system" to scrape metrics related to the system.static_configs
: Configures static targets to scrape.- targets: - localhost
: Prometheus will scrape metrics from the local machine.labels
: Specifies additional labels for the scraped data.job: varlogs
: Identifies this as a job for scraping metrics from/var/log
directories.__path__: /var/log/*log
: Specifies the path pattern to match log files.
Docker Job:
job_name: docker
: Defines a job named "docker" to scrape metrics related to Docker containers.static_configs
: Configures static targets to scrape.- targets: - localhost
: Prometheus will scrape metrics from the local machine.labels
: Specifies additional labels for the scraped data.job: docker_logs
: Identifies this as a job for scraping metrics from Docker containers.__path__: /var/lib/docker/containers/*/*-json.log
: Specifies the path pattern to match Docker container log files.
Before running these 2 configuration files we need to create the docker network loki-net
, run the below command to create:
docker network create loki-net
Before you run the docker-compose.yaml
or any other docker container, you need to do 2 more things:
1. Installing Docker Plugins:
Install the docker drivers for Docker and Grafana/Loki:
docker plugin install grafana/loki-docker-driver:2.9.1 --alias loki --grant-all-permissions
2.Configure Docker daemon.json:
At location /etc/docker/daemon.json
paste the below code:
{
"log-driver": "loki",
"log-opts": {
"loki-url": "http://localhost:3100/loki/api/v1/push",
"loki-batch-size": "400",
}
}
Restart the Docker service:
sudo systemctl restart docker
Run the docker-compose.yaml
:
docker-compose up -d
Running Docker containers:
Run the Docker images of your choice, whose logs you want to monitor. Here I am running the following Docker images:
- Nginx
- Todo
- Reddit-clone
![](https://cdn-images-1.readmedium.com/v2/resize:fit:800/1*KvFtfpe0WrBxmefdon---w.png)
Monitoring Docker Containers logs in Grafana
- Open the Grafana in your web browser running at port number 3000.
If you are opening Grafana for the first time use username: admin and password: admin.
Click on Add your first data source.
![](https://cdn-images-1.readmedium.com/v2/resize:fit:800/1*esbmqY4INmJSexofTSig7g.png)
- Choose Loki, as the data source. Loki is used for connecting the logs of an application.
![](https://cdn-images-1.readmedium.com/v2/resize:fit:800/1*Cm14JoG-KkZV5X3ucyXxvw.png)
- Enter the name and connection URL, scroll down to test and save
![](https://cdn-images-1.readmedium.com/v2/resize:fit:800/1*71NlO3LkHJKc2o874yQpsA.png)
- Click on Add visualization in order to build the dashboard.
![](https://cdn-images-1.readmedium.com/v2/resize:fit:800/1*fOUepPh9Rl8i195o6VlhzQ.png)
- Select the desired data source.
![](https://cdn-images-1.readmedium.com/v2/resize:fit:800/1*fC_L4REz8RvXd8QVbd21Wg.png)
- Select the label,
container_name
select the value, i.e the running container name, let’s start with nginx logs:
![](https://cdn-images-1.readmedium.com/v2/resize:fit:800/1*bBAgUjFJfSSIzc4mUbryrQ.png)
- Select the Logs as the Data visualization tool from the right side
![](https://cdn-images-1.readmedium.com/v2/resize:fit:800/1*c2VUpzDtiFJOLFYu8yb65Q.png)
Click on the run query.
- This query returns the logs generated by the docker container, nginx.
![](https://cdn-images-1.readmedium.com/v2/resize:fit:800/1*DWbtHQ1XKuXH1BGloaK3pA.png)
You can also view logs based on a specific time frame of your choice; currently, it displays all the logs generated in the last 1 hour.
- Now change the
container_name
totodo:
![](https://cdn-images-1.readmedium.com/v2/resize:fit:800/1*Y9s1TNu1UWV_nZ47aRehNQ.png)
Logs generated for todo
container:
![](https://cdn-images-1.readmedium.com/v2/resize:fit:800/1*02f4QeMgR3TBr3o0N30r0A.png)
- Change the
container_name
toredditClone
:
![](https://cdn-images-1.readmedium.com/v2/resize:fit:800/1*Pkm_eXPHJXizI-GtB3v5zw.png)
Logs Generated :
![](https://cdn-images-1.readmedium.com/v2/resize:fit:800/1*tNvcOZVuRFFYFVlvAvoGAQ.png)
Now You can save this Dashboard.
- Click on
Save
![](https://cdn-images-1.readmedium.com/v2/resize:fit:800/1*24Dano81btXWw-hy6O2Dww.png)
- Enter the Dashboard details like name, description and choose the folder to save the dashboard
![](https://cdn-images-1.readmedium.com/v2/resize:fit:800/1*swnbCIKz566X-MJ1fMmyrA.png)
- Your Dashboard has been saved
![](https://cdn-images-1.readmedium.com/v2/resize:fit:800/1*bKkIV-b4blnbnhvvPGETOA.png)
You can change the dashboard, view it, and do some modifications in the dashboard.
Useful links if you get stuck:
- Youtube tutorial for setting up Grafana, Loki, and Promtail using Docker Compose and instaling Docker pluginm configuring: Meet Grafana LOKI, a Log Aggregation System for EVERYTHING
- Grafana Offical Documentation: Docker driver client