avatarPrajwal T

Summary

The web content provides a comprehensive guide on setting up Prometheus for monitoring Kubernetes (K8s) and application-specific metrics within a K8s cluster, including the use of Prometheus Helm charts and the Prometheus Operator, as well as instrumenting a FastAPI application to expose metrics for Prometheus scraping.

Abstract

The article titled "Monitor K8s and Application metrics using Prometheus" delves into the importance of observability in Site Reliability Engineering (SRE) by detailing the process of monitoring Kubernetes and application metrics using Prometheus. It outlines the setup of Prometheus within a K8s cluster, discusses the pros and cons of running Prometheus inside or outside the cluster, and provides step-by-step instructions for deploying Prometheus using Helm charts, specifically the kube-prometheus-stack. The guide also covers the instrumentation of a FastAPI application to expose custom metrics, the configuration of Prometheus to scrape these metrics, and the use of Grafana for visualization. The author emphasizes the importance of understanding the underlying infrastructure and application behavior through logs, metrics, and traces, and concludes by addressing common issues encountered during the setup process.

Opinions

  • The author advocates for running Prometheus inside the K8s cluster for ease of management and integration.
  • The use of Prometheus Helm charts or the Prometheus Operator is recommended over manual setup for better manageability.
  • Instrumenting applications, such as a FastAPI application, is crucial for gaining insights into application-specific metrics.
  • The author suggests that exposing Prometheus and Grafana services using NodePort is a viable option for accessing the UIs outside the K8s cluster, although an Ingress controller is recommended for production environments.
  • The article provides a subjective view that the detailed steps and encountered issues discussed will aid readers in setting up their monitoring stack effectively.

Monitor K8s and Application metrics using Prometheus

To troubleshoot an issue, you must know what is happening under the hood.

Introduction

Observability from SRE (Site Reliability Engineering) terms means, gaining in-depth details about the application and the underlying infrastructure to understand how they operate and how can teams use these details to troubleshoot the issues. The details are mainly from the logs, metrics, and traces.

In this blog, we

  • Set up Prometheus on K8s to collect the K8s-related metrics
  • Instrument a FastAPI application to expose the application-specific metrics

Big Picture

Prometheus

  • Prometheus is an open-source tool that was originally built at SoundCloud for monitoring and alerting.
  • Prometheus primarily deals with “metrics”.
Prometheus Architecture (Source: Offical Prometheus Docs)

For details related to Prometheus architecture please check the official docs: https://prometheus.io/docs/introduction/overview/#architecture

Should I run Prometheus inside or outside the K8s Cluster?

Pros and Cons of running Prometheus inside and outside of the K8s cluster

For this blog, I am installing Prometheus inside the K8s cluster.

Okay, What are some of the ways to set up Prometheus inside the K8s cluster?

  • Manual (AVOID this as it's hard to manage): Create the K8s manifest files for each part of the Prometheus
  • Using Prometheus Helm Chart: Contains Prometheus helm chart. (prometheus-community/prometheus)
  • Using Prometheus Operator: This is an entire Prometheus stack, it comes with Prometheus, Altermanager, Grafana and other components. (prometheus-community/kube-prometheus-stack)

Tools and pre-requisites

Steps to connect K8s and Prometheus

Check my encountered issues before proceeding to the next steps: https://github.com/Prajwalmithun/fastapi-demo-for-medium/issues?q=is%3Aissue+is%3Aclosed

Start minikube cluster

# minikube start --memory 4096

Add the Prometheus repository & update it

# helm repo add prometheus-community https://prometheus-community.github.io/helm-charts

# helm repo list

# helm repo update

Install the Prometheus Helm Charts (with default values) from the repository that I added in the previous step

# helm install prometheus prometheus-community/kube-prometheus-stack

Now, helm has deployed Prometheus on the default K8s namespace. We can confirm by running the following command.

# kubectl get pods

Access the Prometheus UI and Grafana UI

By default, the service type of Prometheus and Grafana is set to “ClusterIP”. If the service type is ClusterIP, we cannot access them outside the K8s cluster.

Possible ways to expose these 2 services to access from outside the cluster:

  1. Port forwarding using kubectl for those pods that run Prometheus and Grafana. (NOT Recommended: Inefficient when there are more than 1 replicas)
  2. Change the service type to “NodePort”. (In this blog)
  3. Ingress controller (Recommended for production)

To change the service type to NodePort, follow these steps:

Verify the service TYPE of “prometheus-kube-prometheus-prometheus” and “prometheus-grafana” by running,

# kubectl get svc prometheus-kube-prometheus-prometheus prometheus-grafana

Run this command, and edit the config of “prometheus-kube-prometheus-prometheus”

# kubectl edit svc prometheus-kube-prometheus-prometheus

Change the config file to look this way. Just add/change those 2 RED highlights.

Now do it similar to “prometheus-grafana”

# k edit svc prometheus-grafana

After changing, your config must look this way.

Verify if the changes are done successfully. If done correctly, the TYPE must be “NodePort”

# kubectl get svc prometheus-kube-prometheus-prometheus prometheus-grafana

Now, we have changed the service type of prometheus and grafana. We can access each of them at these endpoints:

Prometheus

http://<minikube_ip>:30002

Grafana

http://<minikube_ip>:30003

Run this command to know the IP address of minikube node

# minikube ip

Access Prometheus UI

Prometheus UI

In the Prometheus dashboard, n

avigate to “Status -> Targets”. Check if there are entries about the Kubernetes components.

Targets

By running the following expression in the search box, we see the metrics related to the service called “prometheus-grafana”

{service="prometheus-grafana"}

Access the Grafana UI

Grafana UI

Default username/password if deployed via operator is : admin/prom-operator

This is how we can look at the metrics of Kubernetes components in Prometheus. In the next section, we will look at how to instrument an application and monitor the application metrics.

Instrument the application

Instrumenting an application in simple terms means, writing additional code (like importing and using packages, and libraries) to expose the application metrics in a format that tools like Prometheus can understand and scrape it.

The main part of the instrument of this fastAPI is this code.

This exposes an endpoint called “/metrics”. Prometheus will scrape the application-specific metrics from this endpoint.

/metrics

Deploy the application

To deploy the fastAPI to Kubernetes, I have created manifest files. They are under the “k8s-manifest” directory

k8s-manifest

You can clone my repository and run this command to deploy the application.

# git clone https://github.com/Prajwalmithun/fastapi-demo-for-medium.git
# cd fastapi-demo-for-medium/k8s-manifest
# kubectl apply -f deployment.yaml service.yaml servicemonitor.yaml

Access the fastAPI in your browser

http://<minikube_ip>:30001/home

http://<minikube_ip>:30001/all

http://<minikube_ip>:30001/docs

http://<minikube_ip>:30001/metrics

Accessing the app metrics in the Prometheus UI

  • Check under targets. (Status -> Targets). Since there are 2 pods running for the fastAPI, you must see 2 entries.
fastAPI endpoint
  • Also, run a simple query like the following to query all the endpoints that match the label web.
{endpoint="web"}
{endpoint=”web”}

Quick access

Get the IP of the minikube cluster

# minikube ip

API is running on port 30001

http://$(minikube ip):30001

Prometheus is running on port 30002

http://$(minikube ip):30002

Grafana is running on port 30003

http://$(minikube ip):30003

The default username and password for Grafana is

admin/prom-operator

Issues encountered during the setup

Resources

[1] Source code

[2] Minikube installation on Mac M1/M2 chip

[3] Minikube installation on Windows 10 (I haven’t tried on Windows)

[4] Minikube installation on Ubuntu 18.04 / 20.04

[5] Helm installation

Observability
Prometheus
Kubernetes
Monitoring
Grafana
Recommended from ReadMedium
avatarTEJESWAR REDDY
How to Monitor Nginx Logs with Grafana, Loki & Promtail on Docker

8 min read
avatarAkil Blanchard
Monitoring a Kubernetes Cluster using Prometheus and Grafana

In this article we will be looking at deploying Prometheus and Grafana unto a Kubernetes cluster for monitoring and obeservability…

7 min read
avatarsajosam
Easy Kubernetes Monitoring with Prometheus & Grafana (Helm)

As your Kubernetes cluster grows, it’s essential to have a robust monitoring solution in place. Consider a scenario where you have a…

3 min read
avatarAhmad Bilal
Deploying Grafana, Loki, and Prometheus on AWS ECS with EFS and Cloud Formation (Part 3 of 3)

In the first two parts of this series, we explored setting up Grafana, Loki, and Prometheus locally with Docker Compose, followed by…

8 min read
avatarJay
A Guide to Monitoring Kubernetes with Prometheus and Grafana

Introduction

8 min read
avatarMarouane Louguid
Monitoring Docker: Grafana & Prometheus & cAdvisor

Table of Contents

4 min read