How to setup Prometheus, Grafana and node-exporter using Podman, systemd, SSL, and a reverse proxy, with NGINX behind a firewall for monitoring linux servers
TL;DR: Prometheus does not yet work with Podman and Grafana the way I researched it from the documentation. It might be better to set-up both manually. However deploying Grafana with Podman behinda a firewall and a reverse proxy works quite well, so keep consider doing that. It might be better to setup Prometheus manually and not inside a container.
Setup node-exporter
I am using the latest versions as of writing the article. First start by cloning my repo
git clone https://codeberg.org/cap_jmk/prometheus-podFor the initial node-exporter setup run
bash node-exporter.shCreate a service file with
sudo nano /etc/systemd/system/node-exporter.service
And insert (you might replace the path to your executable):
[Unit] Description=Node Exporter [Service] User=nodeexporter ExecStart=path_to_node-exporter_executable/node_exporter Restart=always [Install] WantedBy=multi-user.target
Then run,
sudo systemctl daemon-reload
sudo systemctl start node-exporter.service
sudo systemctl enable node-exporter.serviceDeploy Prometheus and Grafana
bash deploy.sh
Setup NGINX
For more insights about how to setup nginx see my other blogpost.
In your configuration file insert:
server {
listen 6010 http2 ssl;
listen [::]:6010 http2 ssl;
ssl_certificate /etc/ssl/certs/nginx-selfsigned.crt;
ssl_certificate_key /etc/ssl/private/nginx-selfsigned.key;
ssl_dhparam /etc/ssl/certs/dhparam.pem;
server_name your_server;
location / {
proxy_pass http://127.0.0.1:6000;
include proxy_params;
}
}
Setup the Firewall
Please refer to my other article about setting up Gitea about how to setup a the firewall for webapplications.
Take the configuration and just modify the command saying--dport with adding more ports to your webserver (do not delete port 22).
To first list all your rules run
sudo iptables -S
To flush all rules run
sudo iptables -F
Run
sudo iptables -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
sudo iptables -I INPUT -p tcp --dport 80 -j ACCEPT
sudo iptables -I INPUT -p tcp --dport 443 -j ACCEPT
sudo iptables -I INPUT -p tcp --dport 5000 -j ACCEPT
sudo iptables -I INPUT -p tcp --dport 22 -j ACCEPT
sudo iptables -I INPUT -p tcp --dport 6010 -j ACCEPT
sudo iptables -I INPUT -p tcp --dport 9090 -j ACCEPT
sudo iptables -I INPUT -p tcp --dport 9100 -j ACCEPT
sudo iptables -A INPUT -p icmp --icmp-type 0 -m state --state ESTABLISHED,RELATED -j ACCEPT
sudo iptables -A INPUT -i lo -j ACCEPT
sudo iptables -A INPUT -j DROP
sudo iptables -A INPUT -s 192.168.178.0/24 -j ACCEPT #avoid public access
sudo iptables -A INPUT -j REJECT #reject all other IP addressRun
sudo iptables -S
And save the output with
sudo nano /etc/iptables/rules.v4
Make sure you have iptables-persistent installed
sudo apt-get install iptables-persistent
Reload the deamons
sudo systemctl daemon-reload sudo systemctl restart netfilter-persistent.service sudo systemctl status netfilter-persistent.service
Make the configuration persistent
sudo apt-get install iptables-persistent sudo systemctl restart netfilter-persistent.service sudo systemctl status netfilter-persistent.service
Sometimes, it can be useful to delete iptables-persistent and reisntall it.
Test the configuration on localhost:
curl https://localhost:9090 curl https://localhost:9100 curl https://localhost:6000
Then test it on a remote host using your browser, too. You should be done.
Setup Grafana
Please set-up Grafana according to the blog post. For me the fetching of the data does not work reliably. However, for you this might be different.
Conclusion
My conclusion is to run prometheus and node_exporter outside of containers and only host Grafana inside of containers.
For setting up Prometheus on the system level refer to the other blog post.
Join my email list 9k+ and people to learn more about the good lifestyle, technology, and money.
