Monitoring FastAPI Using Grafana and Prometheus

Monitoring APIs is crucial to ensure their health, performance, and reliability. In this guide, we’ll walk through setting up monitoring for a FastAPI application using Prometheus and Grafana. We’ll use Docker Compose to bring up the entire stack: FastAPI, Prometheus, and Grafana.

Step 1: FastAPI Setup

We’ll create a simple FastAPI app with one example endpoint. We’ll use the prometheus_fastapi_instrumentator to collect metrics.

mkdir fastapi-monitoring && cd fastapi-monitoring

Create the app.py file with the following code:

from fastapi import FastAPI
from prometheus_fastapi_instrumentator import Instrumentator
import random

app = FastAPI()

# Instrument the app for Prometheus
Instrumentator().instrument(app).expose(app)

@app.get("/example_endpoint")
async def example_endpoint():
    return {"message": "This is a monitored FastAPI endpoint"}

This FastAPI app has one simple endpoint /example_endpoint that we'll use to demonstrate monitoring.

Step 2: Docker Compose Setup for FastAPI

We’ll set up Docker Compose to bring up FastAPI and other services in one go. Create a docker-compose.yml file in the same directory:

version: '3.8'

services:
  fastapi-app:
    image: tiangolo/uvicorn-gunicorn-fastapi:python3.9
    volumes:
      - ./app.py:/app/app.py
    ports:
      - "80:80"
    environment:
      - PYTHONUNBUFFERED=1

  prometheus:
    image: prom/prometheus:latest
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
    ports:
      - "9090:9090"

  grafana:
    image: grafana/grafana:latest
    ports:
      - "3000:3000"
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=admin

This file sets up three services:

1. FastAPI: Runs our FastAPI app with the /example_endpoint.
2. Prometheus: A Prometheus instance configured to scrape FastAPI metrics.
3. Grafana: A Grafana instance to visualize the metrics.

Step 3: Prometheus Configuration

To enable Prometheus to scrape the FastAPI metrics, we need to configure a prometheus.yml file in the same directory as the Docker Compose file:

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'fastapi'
    metrics_path: /metrics
    static_configs:
      - targets: ['fastapi-app:80']

This configuration tells Prometheus to scrape metrics from FastAPI every 15 seconds.

Step 4: Running the Setup

To run the entire setup, execute:

docker-compose up

This command will bring up FastAPI, Prometheus, and Grafana. You can access the FastAPI app at http://localhost:80/example_endpoint, Prometheus at http://localhost:9090, and Grafana at http://localhost:3000

Step 5: Connecting Prometheus with Grafana

Once the services are up, we’ll need to connect Prometheus as a data source in Grafana:

1. Open Grafana at http://localhost:3000/ and log in using the default credentials (admin/admin).
2. Navigate to Configuration -> Data Sources.
3. Select Prometheus and enter http://prometheus:9090 as the URL.
4. Click Save & Test to confirm the connection.

Now Grafana is connected to Prometheus and can visualize the metrics.

Step 6: Creating Grafana Dashboards

Once you’ve connected Prometheus to Grafana, it’s time to create dashboards for monitoring:

1. Navigate to Grafana at http://localhost:3000
2. Create a new dashboard and add a Graph panel.
3. For each metric, use the following queries:

Total Failed Requests: This query calculates the total number of failed requests (with HTTP status codes 400 or 500) over the last hour. The increase function tracks the increase in the counter metric, while the sum aggregates it.

sum(increase(http_requests_total{status=~"400|500"}[1h]))

Average Query Latency: This query computes the average query latency over the past hour. The rate function calculates the per-second rate of change of both the total request duration (_sum) and the request count (_count), and dividing them gives the average latency.

rate(http_request_duration_seconds_sum[1h]) / rate(http_request_duration_seconds_count[1h])

Request Count by Endpoints: This query sums the total number of requests made to the /example_endpoint over the last hour. It uses increase to calculate the growth in requests and aggregates the count using sum.

sum(increase(http_requests_total{handler="/example_endpoint"}[1h]))

Step 7: Setting Alerts in Grafana

To set up alerts for failed requests exceeding a threshold, follow these steps:

1. In your Grafana dashboard, edit the Total Failed Requests panel.
2. Click on the Alert tab, and configure the alert as follows:

• Conditions: When the query result is above 10
• Evaluation Period: Every 1m for 1h (Grafana will check the condition once every minute and look at the data from the past hour to determine if the alert should be triggered)

3. Save the alert.

This will trigger an alert if failed requests within an hour exceed 10.

Conclusion

By following this guide, you can monitor any FastAPI app using Prometheus and Grafana. You’ll have insights into failed requests, response times, and traffic for individual endpoints. With the addition of alerts, you’ll always be notified when issues arise. This monitoring stack helps maintain the reliability and performance of your APIs, ensuring smooth operations.