Setting Up ELK Stack on Kubernetes: A Step-by-Step Guide
In the world of modern DevOps, the ELK stack (Elasticsearch, Logstash, and Kibana) has become an essential tool for log management and analysis. Combining the power of ELK with the scalability of Kubernetes (k8s) can significantly enhance your logging capabilities. This guide will walk you through the process of setting up an ELK stack on a Kubernetes cluster.
Table of Contents
- Introduction
- Prerequisites
- Setting Up Elasticsearch
- Setting Up Kibana
- Deploying Filebeat
- Deploying Logstash
- Configuring and Launching Kibana
1. Introduction
In today’s world of microservices and distributed systems, effective logging and monitoring are crucial for maintaining the health and performance of your applications. The ELK stack — comprising Elasticsearch, Logstash, and Kibana — along with Filebeat, provides a powerful solution for aggregating and analyzing log data. In this blog, we will walk through the steps to set up the ELK stack with Filebeat on a Kubernetes cluster.
2. Prerequisites
Before we begin, ensure you have the following:
- A running Kubernetes cluster (Minikube, EKS, GKE, etc.).
- kubectl command-line tool configured to interact with your cluster.
- Lens software to connect and see all the information related to the cluster.
Install Operator
Things to consider before you start:
Install custom resource definitions and Install the operator with its RBAC rules:
kubectl create -f https://download.elastic.co/downloads/eck/2.2.0/crds.yaml kubectl apply -f https://download.elastic.co/downloads/eck/2.2.0/operator.yaml
{MAKE SURE ALL YOUR RESOURCES ARE OF THE SAME VERSION THAT IS ELASTICSEARCH, KIBANA, LOGSTASH & FILEBEAT. }
3: Setting Up Elasticsearch
First, we’ll deploy Elasticsearch, which will store our log data. For this I am using a YAML file in which I have mention all the details for the Elasticsearch called “elasticsearch.yaml” as below.
apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
name: quickstart
spec:
version: 8.13.4
nodeSets:
- name: default
count: 1
config:
node.store.allow_mmap: falseAfter creating the script file we have to apply this YAML file by applying the below command on Lens or Terminal
kubectl apply -f elasticsearch.yamlTo monitor cluster health and creation progress apply this command:
kubectl get elasticsearch
Request Elasticsearch access
A ClusterIP Service is automatically created for your cluster:
kubectl get service quickstart-es-http3.1: Get the credentials.
A default user named elastic is automatically created with the password stored in a Kubernetes secret:
PASSWORD=$(kubectl get secret quickstart-es-elastic-user -o go-template='{{.data.elastic | base64decode}}')3.2: Request the Elasticsearch endpoint.
From inside the Kubernetes cluster:
curl -u "elastic:$PASSWORD" -k "https://quickstart-es-http:9200"From your local workstation, use the following command in a separate terminal:
kubectl port-forward service/quickstart-es-http 9200
Then request localhost:
curl -u "elastic:$PASSWORD" -k "https://localhost:9200"Or Open https://localhost:9200 in your browser. Your browser will show a warning because the self-signed certificate configured by default is not verified by a known certificate authority and not trusted by your browser. You can temporarily acknowledge the warning for the purposes of this quick start but it is highly recommended that you configure valid certificates for any production deployments.
Login as the elastic user and password is stored in lens (Your Cluster) -> Config -> Secrets -> (keys = elastic).
4: Setting Up Kibana
Kibana provides a web interface to visualize logs stored in Elasticsearch. For that create a Kibana instance and associate it with your Elasticsearch cluster and for that create a YAML file “kibana.yaml” as mentioned below
apiVersion: kibana.k8s.elastic.co/v1
kind: Kibana
metadata:
name: quickstart
spec:
version: 8.13.4
count: 1
elasticsearchRef:
name: quickstart
# http:
# tls:
# selfSignedCertificate:
# disabled: trueAfter creating the script file we have to apply this YAML file by applying the below command on Lens terminal:
kubectl apply -f kibana.yaml4.1: Monitor Kibana health and creation progress.
Similar to Elasticsearch, you can retrieve details about Kibana instances:
kubectl get kibana4.2: To access Kibana.
A ClusterIP Service is automatically created for Kibana:
kubectl get service quickstart-kb-httpUse kubectl port-forward to access Kibana from your local workstation:
kubectl port-forward service/quickstart-kb-http 5601
Open https://localhost:5601 in your browser. Your browser will show a warning because the self-signed certificate configured by default is not verified by a known certificate authority and not trusted by your browser. You can temporarily acknowledge the warning for the purposes of this quick start but it is highly recommended that you configure valid certificates for any production deployments.
Login as the elastic user. The password can be obtained with the following command:
kubectl get secret quickstart-es-elastic-user -o=jsonpath='{.data.elastic}' | base64 --decode; echo5: Deploying Filebeat
Filebeat is a lightweight shipper for forwarding and centralizing log data. Apply the following specification to deploy Filebeat and collect the logs of all containers running in the Kubernetes cluster.
For that create a filebeat instance and for that create a YAML file “filebeat.yaml” as mentioned below:
apiVersion: beat.k8s.elastic.co/v1beta1
kind: Beat
metadata:
name: quickstart
spec:
type: filebeat
version: 8.13.4
# elasticsearchRef:
# name: quickstart
# kibanaRef:
# name: kibana
config:
filebeat.inputs:
- type: container
paths:
- /var/log/containers/*.log
processors:
- add_kubernetes_metadata:
host: ${NODE_NAME}
matchers:
- logs_path:
logs_path: "/var/log/containers/"
- drop_event.when:
or:
- equals:
kubernetes.namespace: "kube-system"
- equals:
kubernetes.namespace: "kube-public"
- equals:
kubernetes.namespace: "quickstart"
- equals:
kubernetes.namespace: "kube-node-lease"
- equals:
kubernetes.namespace: "elastic-system"
output.logstash:
hosts: ["logstash.logging.svc:5044"]
daemonSet:
podTemplate:
spec:
serviceAccountName: filebeat
automountServiceAccountToken: true
terminationGracePeriodSeconds: 30
tolerations:
- key: dedicated
operator: Exists
effect: NoSchedule
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true # Allows to provide richer host metadata
containers:
- name: filebeat
securityContext:
runAsUser: 0
# If using Red Hat OpenShift uncomment this:
#privileged: true
volumeMounts:
- name: varlogcontainers
mountPath: /var/log/containers
- name: varlogpods
mountPath: /var/log/pods
- name: varlibdockercontainers
mountPath: /var/lib/docker/containers
env:
- name: NODE_NAME
valueFrom:
fieldRef:
fieldPath: spec.nodeName
resources:
limits:
cpu: 500m
memory: 2000Mi
requests:
cpu: 100m
memory: 200Mi
volumes:
- name: varlogcontainers
hostPath:
path: /var/log/containers
- name: varlogpods
hostPath:
path: /var/log/pods
- name: varlibdockercontainers
hostPath:
path: /var/lib/docker/containers
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: filebeat
rules:
- apiGroups: [""] # "" indicates the core API group
resources:
- namespaces
- pods
- nodes
verbs:
- get
- watch
- list
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: filebeat
subjects:
- kind: ServiceAccount
name: filebeat
namespace: elk-test
roleRef:
kind: ClusterRole
name: filebeat
apiGroup: rbac.authorization.k8s.io
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: filebeatAfter creating the script file we have to apply this YAML file by applying the below command on Lens terminal:
kubectl apply -f filebeat.yaml5.1: Monitor Beats.
Retrieve details about the Filebeat.
kubectl get beat5.2: Access logs of the filebeat Pods.
kubectl logs -f quickstart-beat-filebeat-plr8h
5.3: Access logs ingested by Filebeat.
You have two options:
- Follow the Elasticsearch deployment guide and run:
curl -u "elastic:$PASSWORD" -k "https://localhost:9200/filebeat-*/_search"- Follow the Kibana deployment guide, log in and go to Kibana > Discover.
6: Deploying Logstash
Add the following specification to create a minimal Logstash deployment that will listen to a Beats agent or Elastic Agent configured to send to Logstash on port 5044, create the service and write the output to an Elasticsearch cluster named quickstart, created in the Elasticsearch quickstart.
For that create a logstash instance and for that create a YAML file “logstash.yaml” as mentioned below:
apiVersion: apps/v1
kind: Deployment
metadata:
name: logstash
labels:
app.kubernetes.io/name: elasticsearch-logstash
app.kubernetes.io/component: logstash
spec:
replicas: 1
selector:
matchLabels:
app.kubernetes.io/name: elasticsearch-logstash
app.kubernetes.io/component: logstash
template:
metadata:
labels:
app.kubernetes.io/name: elasticsearch-logstash
app.kubernetes.io/component: logstash
spec:
containers:
- name: logstash
image: docker.elastic.co/logstash/logstash:8.13.4
ports:
- name: "tcp-beats"
containerPort: 5044
env:
- name: ES_HOSTS
value: "https://quickstart-es-http.elk-test.svc:9200"
- name: ES_USER
value: "elastic"
- name: ES_PASSWORD
valueFrom:
secretKeyRef:
name: quickstart-es-elastic-user
key: elastic
volumeMounts:
- name: config-volume
mountPath: /usr/share/logstash/config
- name: pipeline-volume
mountPath: /usr/share/logstash/pipeline
# - name: ca-certs
# mountPath: /etc/logstash/certificates
# readOnly: true
volumes:
- name: config-volume
configMap:
name: logstash-config
- name: pipeline-volume
configMap:
name: logstash-pipeline
# - name: ca-certs
# secret:
# secretName: elasticsearch-es-http-certs-public
---
apiVersion: v1
kind: Service
metadata:
name: logstash
labels:
app.kubernetes.io/name: elasticsearch-logstash
app.kubernetes.io/component: logstash
spec:
ports:
- name: "tcp-beats"
port: 5044
targetPort: 5044
selector:
app.kubernetes.io/name: elasticsearch-logstash
app.kubernetes.io/component: logstashThe config maps defined in the following YAML file are given as volume in the logstash deployment yaml file named as “logstash-config.yaml” as mentioned below:
apiVersion: v1
kind: ConfigMap
metadata:
name: quickstart
labels:
app.kubernetes.io/name: elasticsearch-logstash
app.kubernetes.io/component: logstash
data:
logstash.yml: |
http.host: 0.0.0.0
pipeline.ecs_compatibility: disabled
pipelines.yml: |
- pipeline.id: logstash
path.config: "/usr/share/logstash/pipeline/logstash.conf"
log4j2.properties: |
logger.logstashpipeline.name = logstash.inputs.beats
logger.logstashpipeline.level = error
---
apiVersion: v1
kind: ConfigMap
metadata:
name: logstash-pipeline
labels:
app.kubernetes.io/name: elasticsearch-logstash
app.kubernetes.io/component: logstash
data:
logstash.conf: |
input {
beats {
port => 5044
}
}
filter {
json {
source => "message"
}
prune {
whitelist_names => [ "msg" ]
}
mutate {
rename => { "msg" => "message" }
}
}
output {
if [message] =~ "admission" {
elasticsearch {
index => "logstashadmission-%{+YYYY.MM.dd}"
hosts => [ "${ES_HOSTS}" ]
user => "${ES_USER}"
password => "${ES_PASSWORD}"
cacert => '/etc/logstash/certificates/ca.crt'
}
}
}After creating the script file we have to apply this YAML file by applying the below command on Lens terminal:
kubectl apply -f logstash.yaml
kubectl apply -f logstash-config.yaml6.1: Check the status of Logstash
kubectl get logstash7: Accessing and Visualizing Logs in Kibana
Once Filebeat is running and shipping logs, you can access Kibana to visualize the logs.
- Navigate to the Discover tab in Kibana.
- Select the Filebeat index pattern (you may need to create this pattern if it’s your first time).
- Explore your logs!
Conclusion
Setting up the ELK stack with Filebeat on a Kubernetes cluster can greatly enhance your ability to monitor and troubleshoot your applications. With Elasticsearch storing your logs, Filebeat shipping them, and Kibana providing visualization, you have a powerful toolset at your disposal.
Thank you for reading….❤️❤️ I hope you learn something new.