Setup Grafana Loki Basic Auth Behind Nginx Reverse Proxy (HTTPS)
Quick Introduction
Grafana Loki, a powerful and scalable log aggregation system, lacks built-in authentication. To protect your log data, it’s essential to implement a robust authentication mechanism. This post guides you through setting up basic authentication for Loki behind an Nginx reverse proxy.
Promtail is a log collection agent designed to efficiently gather log data from various sources and send it to a Grafana Loki instance for storage and analysis. It’s a crucial component of the Grafana Loki stack, working alongside Loki and Grafana to create a comprehensive log management solution.
Monitoring and logging are crucial aspects of maintaining the health and performance of your applications. Grafana Loki, a powerful tool for log aggregation and visualization, simplifies this task. By setting it up behind an Nginx reverse proxy with basic authentication, you can secure your logging infrastructure efficiently. This guide will walk you through the entire process step by step.
Why Use Nginx as a Reverse Proxy?
Nginx is a high-performance web server and reverse proxy renowned for its efficiency and flexibility. By placing Nginx in front of Loki, you can:
- Enhance security: Implement basic authentication to restrict access to your log data.
- Load balancing: Distribute traffic across multiple Loki instances.
- SSL termination: Secure communication with HTTPS.
- Caching: Improve performance by caching static content.
Prerequisites
Required Software and Tools
Before we begin, ensure you have the following:
- A Linux server (Ubuntu/Debian preferred)
- Root or sudo access
- Grafana Loki and Grafana binaries
- Nginx installed
- Basic understanding of Linux command-line operations
A.) Loki Installation and Configuration
- Add Grafana Repository
# mkdir -p /etc/apt/keyrings/
# wget -q -O - https://apt.grafana.com/gpg.key | gpg --dearmor > /etc/apt/keyrings/grafana.gpg
# echo "deb [signed-by=/etc/apt/keyrings/grafana.gpg] https://apt.grafana.com stable main" | tee /etc/apt/sources.list.d/grafana.list2. Install Loki
apt-get update
apt-get install loki
systemd start loki && systemd enable loki
systemd status loki3. Configure Loki Configuration File (Example)
mv /etc/loki/local-config.yaml /etc/loki/local-config.yaml.bak
sudo nano /etc/loki/local-config.yamlauth_enabled: false
#By default loki running on port 3100
server:
http_listen_port: 3100
ingester:
lifecycler:
address: 127.0.0.1
ring:
kvstore:
store: inmemory
replication_factor: 1
final_sleep: 0s
chunk_idle_period: 1h # Any chunk not receiving new logs in this time will be flushed
max_chunk_age: 1h # All chunks will be flushed when they hit this age, default is 1h
chunk_target_size: 1048576 # Loki will attempt to build chunks up to 1.5MB, flushing first if chunk_idle_period or max_chunk_age is reached first
chunk_retain_period: 30s # Must be greater than index read cache TTL if using an index cache (Default index read cache TTL is 5m)
max_transfer_retries: 0 # Chunk transfers disabled
schema_config:
configs:
- from: 2020-10-24
store: boltdb-shipper
object_store: filesystem
schema: v11
index:
prefix: index_
period: 24h
storage_config:
boltdb_shipper:
active_index_directory: /loki/boltdb-shipper-active
cache_location: /loki/boltdb-shipper-cache
cache_ttl: 24h # Can be increased for faster performance over longer query periods, uses more disk space
shared_store: filesystem
filesystem:
directory: /loki/chunks
compactor:
working_directory: /loki/boltdb-shipper-compactor
shared_store: filesystem
limits_config:
reject_old_samples: true
reject_old_samples_max_age: 168h
ingestion_burst_size_mb: 16
ingestion_rate_mb: 16
chunk_store_config:
max_look_back_period: 0s
table_manager:
retention_deletes_enabled: true
retention_period: 90d # Set retention period to 90 days
ruler:
storage:
type: local
local:
directory: /loki/rules
rule_path: /loki/rules-temp
alertmanager_url: localhost
ring:
kvstore:
store: inmemory
enable_api: true4. Install and Configure Nginx
apt-get install nginx
5. Create loki nginx config
sudo nano /etc/nginx/sites-available/<LOKI_DOMAIN_NAME>.conf
server {
listen 80;
server_name <LOKI_DOMAIN_NAME>; #Repalce with your domain name
location / {
proxy_pass http://localhost:3100; #match with running loki port number
proxy_set_header Host \$host;
proxy_set_header X-Real-IP \$remote_addr;
proxy_set_header X-Forwarded-For \$proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto \$scheme;
}
}
server {
listen 80;
server_name <LOKI_DOMAIN_NAME>; #Repalce with your domain name
location / {
proxy_read_timeout 1800s;
proxy_connect_timeout 1600s;
proxy_pass http://localhost:3100; #match with running loki port number
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $connection_upgrade;
proxy_set_header Connection "Keep-Alive";
proxy_set_header Proxy-Connection "Keep-Alive";
proxy_redirect off;
}
location /ready {
proxy_pass http://localhost:3100; #match with running loki port number
proxy_http_version 1.1;
proxy_set_header Connection "Keep-Alive";
proxy_set_header Proxy-Connection "Keep-Alive";
proxy_redirect off;
auth_basic "off";
}
}sudo ln -s /etc/nginx/sites-available/<LOKI_DOMAIN_NAME>.conf /etc/nginx/sites-enabled/7. Configure nginx HTTPS with certbot
sudo snap install --classic certbot
sudo ln -s /snap/bin/certbot /usr/bin/certbot
sudo certbot --nginx
8. Setup Loki HTTP Basic Auth
sudo apt-get install apache2-utils
sudo htpasswd -c /etc/nginx/.loki_htpasswd <LOKI_USERNAME>
>>> Enter Password <LOKI_PASSWORD>9. Configure Loki HTTP Basic Auth Nginx Config File
server {
server_name <LOKI_DOMAIN_NAME>;
auth_basic "loki auth";
auth_basic_user_file /etc/nginx/.loki_htpasswd;
location / {
proxy_read_timeout 1800s;
proxy_connect_timeout 1600s;
proxy_pass http://localhost:3100;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $connection_upgrade;
proxy_set_header Connection "Keep-Alive";
proxy_set_header Proxy-Connection "Keep-Alive";
proxy_redirect off;
}
location /ready {
proxy_pass http://localhost:3100;
proxy_http_version 1.1;
proxy_set_header Connection "Keep-Alive";
proxy_set_header Proxy-Connection "Keep-Alive";
proxy_redirect off;
auth_basic "off";
}
listen 443 ssl; # managed by Certbot
ssl_certificate /etc/letsencrypt/live/<LOKI_DOMAIN_NAME>/fullchain.pem; # managed by Certbot
ssl_certificate_key /etc/letsencrypt/live/<LOKI_DOMAIN_NAME>/privkey.pem; # managed by Certbot
include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot
}
server {
if ($host = <LOKI_DOMAIN_NAME>) {
return 301 https://$host$request_uri;
} # managed by Certbot
listen 80;
server_name <LOKI_DOMAIN_NAME>;
return 404; # managed by CertbotB.) Promtail Installation and Configuration
- Add Grafana Repository
# mkdir -p /etc/apt/keyrings/
# wget -q -O - https://apt.grafana.com/gpg.key | gpg --dearmor > /etc/apt/keyrings/grafana.gpg
# echo "deb [signed-by=/etc/apt/keyrings/grafana.gpg] https://apt.grafana.com stable main" | tee /etc/apt/sources.list.d/grafana.list2. Install Promtail
apt-get update
apt-get install promtail
systemd start promtail && systemd enable promtail
systemd status promtail3. Configure Promtail Configuration File (Example)
mv /etc/promtail/promtail.yaml /etc/promtail/promtail.yaml.bak
sudo nano /etc/promtail/promtail.yamlserver:
http_listen_port: 9080
grpc_listen_port: 0
positions:
filename: /tmp/positions.yaml
#Replace loki username, password and domain name
clients:
- url: https://<LOKI_DOMAIN_NAME>/loki/api/v1/push
basic_auth:
username: <LOKI_USERNAME>
password: <LOKI_PASSWORD>
scrape_configs:
- job_name: system
static_configs:
- targets:
- localhost
labels:
job: varlogs
__path__: /var/log/*log
stream: stdoutC.) Configuration Test
For testing the configuration is correct you can use curl command inside promtail instance
curl -i -u <LOKI_USERNAME>:<LOKI_PASSWORD> https://<LOKI_DOMAIN_NAME>/loki/api/v1/labelsIf everything is setup correctly you should see this output below:
HTTP/1.1 200 OK
Server: nginx/1.18.0 (Ubuntu)
Date: Tue, 06 Aug 2024 08:24:20 GMT
Content-Type: application/json; charset=UTF-8
Content-Length: 71
Connection: keep-alive
Vary: Accept-Encoding
{"status":"success","data":["filename","job","service_name","stream"]}Conclusion
By following these steps, you’ve successfully secured your Grafana Loki instance with basic authentication behind an Nginx reverse proxy. This provides a solid foundation for protecting your log data.
Reference:
