Summary

The website content provides a guide on setting up an observability stack with Microk8s, detailing the installation process, configuration, and usage of the Grafana/Prometheus/Loki stack for monitoring Kubernetes clusters.

Abstract

The article titled "Observability With Microk8s" introduces readers to the built-in observability features of Microk8s, emphasizing the joy of log monitoring for service management. It outlines the necessary steps to enable the observability stack using a single command and highlights the importance of noting down the generated username and password for later use. The author guides the reader through the process of accessing Grafana's UI via port forwarding and provides instructions on how to navigate and utilize the pre-configured data sources: Prometheus for metrics, Tempo for traces, and Loki for logs. The article includes practical examples, such as filtering out health-check logs from a React frontend service and creating custom dashboards for real-time monitoring. The author encourages celebrating small successes, like successfully setting up the observability stack, and concludes by inviting readers to explore the pre-created dashboards for insights into their Kubernetes pods' performance without the need for additional service modifications.

Opinions

The author expresses a personal enthusiasm for log monitoring, identifying as a "log geek."
The author prefers using k9s over Lens for its simplicity when managing Kubernetes clusters.
A critical warning regarding CPU usage is mentioned, suggesting that users may require a more powerful server if the issue persists.
The author recommends celebrating small successes, such as setting up the observability stack, to maintain motivation.
The article subtly promotes an AI service, ZAI.chat, as a cost-effective alternative to ChatGPT Plus (GPT-4), offering a special subscription rate.

Observability With Microk8s

Easy logging for your Kubernetes cluster

I’m a real log geek; I love watching logs scroll by as my services run. So I was very happy to learn that Microk8s has an observability stack built-in. But it still requires some set-up and finessing, which I will review in this article.

To follow along, you should have microk8s installed, and I will use k9s for cluster visibility. You can use Lens, but I like k9s for its simplicity. In my Kubernetes cluster, I have a few microservices and databases running. Going on k9s, I see this:

To install the observability stack, you can run this command:

microk8s enable observability

It will print out the status as it installs several pieces. Jot down the username/password it displays, which you will use later. Now, you can look at the observability namespace in k9s:

Notice the Critical CPU level warning. That goes away after a while. If it doesn’t, you may need a beefier server.

Highlight the kube-prom-stack-grafana pod and port forward the 3000 port with the shift-f key

That should expose the Grafana UI. You can browse to http://localhost:3000 and you’ll see this:

Put in the username/password you saved earlier to get to the main dashboard:

If you managed to get this far without issue, reward yourself with a cup of coffee! Celebrate the small successes; there’s more work ahead.

Once you finish your coffee, it’s time to get back to work. Three data sources are configured with the microk8s observability module. They are Prometheus, Tempo, and Loki. Prometheus is a metrics monitor, Tempo works with traces across services, and Loki reads the logs. We’ll look at Loki first. From the main menu, select Explore.

You’ll get to the query builder. I happen to have a service blackbook-ui that is a simple Nginx server for a React front end. So under the label field, I select ‘App,’ and in the value field, I can pick the Blackbook UI service from the drop-down. From there, I can see the logs, mostly Kubernetes health-check pings.

To remove these, I can add an operation of “Line Does Not Contain” and set the value to kube-probe.

Now, I can see actual requests being logged. Select the Add to Dashboard button to create a new dashboard. Set the titles and set refresh at ten seconds, and voila!

Back at the main menu, you can select dashboards and see your dashboard among the pre-created ones.

Next, we can look at some of the pre-created dashboards. Pick the Kubernetes / Compute Resources / Namespace (Pods) dashboards and select the namespace you want to look at. This will show metrics from Prometheus for the pods in the selected namespace:

All this without changing or adding to the running services.

This should get you up and running with the Grafana/Prometheus/Loki stack. You can now embrace your inner log geek and let your curiosity run wild!