Mastering Kubernetes One Task at a Time — DaemonSet Demonstration
A simple file cleanup utility showing how to write Kubernetes DaemonSets and perform rolling updates and rollbacks.
Table of Contents
- Introduction
- Kubernetes DaemonSet Resource References
- Kubernetes Clusters for Hands-On Practice
- Download DaemonSets Demonstration Code from GitHub
- OBJECTIVE 1 — A File Cleanup Utility DaemonSet Running on All Nodes
- OBJECTIVE 2 — Performing a Rolling Update on a DaemonSet
- OBJECTIVE 3 — Performing a Rollback on a DaemonSet
- Final Cleanup
- Conclusion
Introduction
One of my earlier posts, Mastering Kubernetes One Task at a Time — Pods, Deployments, and ReplicaSets, covered the first of the four Kubernetes built-in Workload Resources groups. Visit the link below for a quick recap.
Subsequently, I also wrote two posts on Kubernetes Volumes, showing how to implement:
Now, we're ready to apply the knowledge garnered from the reading and practice thus far to implement the next very important Kubernetes workload resource, DaemonSet.
This post demonstrates the following three objectives—
- How to write a DaemonSet with an example file clean-up utility that runs on each node of a Kubernetes cluster, including the control plane node.
- How to perform a rolling update on a DaemonSet
- How to perform a rollback on a DaemonSet
Kubernetes DaemonSet Resource References
For this post, I will refer to the following Kubernetes.io documentation pages.
- https://kubernetes.io/docs/concepts/workloads/
- https://kubernetes.io/docs/concepts/workloads/controllers/daemonset
- https://kubernetes.io/docs/tasks/manage-daemon/rollback-daemon-set/
- https://kubernetes.io/docs/tasks/manage-daemon/update-daemon-set/
- https://kubernetes.io/docs/reference/kubernetes-api/workload-resources/daemon-set-v1/
Worth highlighting here is the following for a quick understanding.
A DaemonSet ensures that all (or some) Nodes run a copy of a Pod. As nodes are added to the cluster, Pods are added to them. As nodes are removed from the cluster, those Pods are garbage collected. Deleting a DaemonSet will clean up the Pods it created.
Some typical uses of a DaemonSet are:
. running a cluster storage daemon on every node
. running a logs collection daemon on every node
. running a node monitoring daemon on every node
Kubernetes Clusters for Hands-On Practice
This DaemonSet demonstration works on any Kubernetes cluster, including the following two clusters that you're free to spin up anytime in a matter of minutes to practice all your Kubernetes hands-on exercises.
I am demonstrating on the second one below, the Kubeadm-based cluster.
Download DaemonSets Demonstration Code from GitHub
Let's download the code repo and walk through the THREE OBJECTIVES step by step while executing the code along the way.
# Clone the jdluther-kubernetes-io-tasks github repo
$ REPO=jdluther-kubernetes-io-tasks && \
git clone https://github.com/jdluther2020/$REPO.git && \
cd $REPO
$ cd daemonsets-demonstration && ls -al
# Check the code files
$ tree
.
├── daemonsets-demonstration.sh
├── file-cleaner-all-nodes.yaml
└── web-app-all-nodes.yaml
# daemonsets-demonstration.sh script covers the entire demonstration.
# This is how the demonstration Kubernetes cluster looks like
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
ip-172-31-0-221.ec2.internal Ready control-plane 30m v1.27.1
ip-172-31-13-253.ec2.internal Ready <none> 27m v1.27.1
ip-172-31-4-118.ec2.internal Ready <none> 27m v1.27.1OBJECTIVE 1 — A File Cleanup Utility DaemonSet Running on All Nodes
Having access to all the code files in the daemonsets-demonstration code folder, let's execute them step by step.
The purpose of this DaemonSet demonstration is to show that the daemon running on each node wakes up every 60 seconds and deletes the old files inside the /var/web/data folder written by the web app, also running on each node, in 10-second intervals.
STEP 1 — Install the web app deployment
# Quickly browse the web app deployment manifest
# We have three replicas, expecting scheduling of a pod on each node.
# The app creates a whole bunch of files on the hostPath of each node
$ cat web-app-all-nodes.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: web-app
spec:
replicas: 3
selector:
matchLabels:
app: web-app
template:
metadata:
labels:
app: web-app
spec:
tolerations:
- key: node-role.kubernetes.io/control-plane
operator: Exists
effect: NoSchedule
- key: node-role.kubernetes.io/master
operator: Exists
effect: NoSchedule
volumes:
- name: host-vol
hostPath:
path: /var/web/data
type: DirectoryOrCreate
containers:
- name: web-app-container
image: nginx:stable
command: ["/bin/sh", "-c"]
args:
- >
count=0;
while true;
do
file="/data/file-$count.txt";
echo "`date` : Written by pod $POD_NAME running on $NODE_NAME"\
> $file;
sleep 10;
count=`expr $count + 1`
done
volumeMounts:
- name: host-vol
mountPath: /data
env:
- name: NODE_NAME
valueFrom:
fieldRef:
fieldPath: spec.nodeName
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
# Deploy the web app
kubectl create -f web-app-all-nodes.yaml
# Check to make sure pods are running and their respective hosts
$ kubectl get pods -o wide
NAME READY STATUS NODE ...
web-app-75f6bd9b69-fnfkw 1/1 Running ip-172-31-4-118.ec2.internal ...
web-app-75f6bd9b69-lk24l 1/1 Running ip-172-31-0-221.ec2.internal ...
web-app-75f6bd9b69-wsnfd 1/1 Running ip-172-31-13-253.ec2.internal...
# Check the data files written by the deployed app
# Expect to see a bunch of files sicne a file is written every 10 seconds
# Use appropriate pod name from kubectl get pods command output above
kubectl exec -it web-app-75f6bd9b69-fnfkw -- ls -ltr /dataSTEP 2— Deploy the file clean-up utility DaemonSet
# Review the DaemonSet manifest
$ cat file-cleaner-all-nodes.yaml
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: file-cleaner
spec:
selector:
matchLabels:
app: file-cleaner
updateStrategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 1
template:
metadata:
labels:
app: file-cleaner
spec:
tolerations:
- key: node-role.kubernetes.io/control-plane
operator: Exists
effect: NoSchedule
#- key: node-role.kubernetes.io/master
# operator: Exists
# effect: NoSchedule
containers:
- name: file-cleaner-container
image: busybox:stable
command: ['sh', '-c', 'while true; do rm `ls -t /data/* |\
awk "NR>5"`; ls -l /data; sleep 60; done']
volumeMounts:
- name: host-vol
mountPath: /data
volumes:
- name: host-vol
hostPath:
path: /var/web/data
type: Directory
# Deploy the DaemonSet, responsible to clean up all except the last 5 files
kubectl create -f file-cleaner-all-nodes.yaml
# Verify the daemonset pods are running, one in each node
$ NAME READY STATUS NODE ...
file-cleaner-n4pnt 1/1 Running ip-172-31-0-221.ec2.internal ...
file-cleaner-p82r6 1/1 Running ip-172-31-13-253.ec2.internal ...
file-cleaner-q2k2m 1/1 Running ip-172-31-4-118.ec2.internal ...
# Verify the daemon script is doing its job by looking at the pod log
# Expect to see the last 5 files remaining, written every 60 seconds
kubectl logs file-cleaner-q2k2mNote that since we intended to run the DaemonSet on all nodes, including the control plane, the DaemonSet manifest included the tolerations block above. Without it, the DaemonSet will run on worker nodes only. Refer to the documentation page Running Pods on Only Some Nodes to control DaemonSet hosts via node names and node labels.
Let's quickly examine the taints field in the node description so that you can relate it to the tolerations in the DaemonSet manifest.
# Check for taints in the node description
# Note: Some nodes may have multiple taints
$ kubectl describe nodes | grep -i -A1 taint
Taints: node-role.kubernetes.io/control-plane:NoSchedule
...
--
Taints: <none>
...
--
Taints: <none>
...OBJECTIVE 2 — Performing a Rolling Update on a DaemonSet
DaemonSet has two update strategy types, OnDelete and RollingUpdate, the latter being the default. See the documentation for more details.
Let's watch it in action.
# Confirm the current Daemonset update strategy
$ kubectl get ds/file-cleaner \
-o go-template='{{.spec.updateStrategy.type}}{{"\n"}}'
# Any updates to the .spec.template section triggers a rolling update
# (e.g. container image, resources etc.)
# Updates can take place by
# 1. Editing the Daemonset object live with edit command or
# 2. Changing/re-applying the manifest or
# 3. Imperative command modification
# Let's change the image imperatively and see the rolling update process
$ kubectl set image ds/file-cleaner file-cleaner-container=busybox:latest
# Watch the rolling update rollout status of the latest DaemonSet
$ kubectl rollout status ds/file-cleaner
Waiting for daemon set "file-cleaner" rollout to finish: 0 out of 3 new ...
Waiting for daemon set "file-cleaner" rollout to finish: 2 of 3 updated ...
daemon set "file-cleaner" successfully rolled outOBJECTIVE 3 — Performing a Rollback on a DaemonSet
Knowing how to rollback is helpful if, for some reason, the update runs into an error or we need to rollback a successful update.
Here are the steps to perform a rollback.
# Start by listing all revisions of a DaemonSet update
$ kubectl rollout history ds/file-cleaner
daemonset.apps/file-cleaner
REVISION CHANGE-CAUSE
1 <none>
2 <none>
# Check the details of a specific revision
$ kubectl rollout history ds/file-cleaner --revision=1
daemonset.apps/file-cleaner with revision #1
Pod Template:
Labels: app=file-cleaner
Containers:
file-cleaner-container:
Image: busybox:stable
Port: <none>
Host Port: <none>
Command:
sh
-c
while true; do rm `ls -t /data/* | awk "NR>5"`;\
ls -l /data; sleep 60; done
Environment: <none>
Mounts:
/data from host-vol (rw)
Volumes:
host-vol:
Type: HostPath (bare host directory volume)
Path: /var/web/data
HostPathType: Directory
# Roll back to a specific revision
# Without the optional --to-revision, it reverts to the previous revision
$ kubectl rollout undo ds/file-cleaner --to-revision=1
daemonset.apps/file-cleaner rolled back
# The rollback increments the revision
$ kubectl rollout history ds/file-cleaner
daemonset.apps/file-cleaner
REVISION CHANGE-CAUSE
2 <none>
3 <none>Final Cleanup
At this point, we're done deploying the necessary components and demonstrating and verifying the DaemonSet functionalities.
It's time to clean up the resources.
# List the resources running
$ kubectl get deploy,ds,pods
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/web-app 3/3 3 3 122m
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
daemonset.apps/file-cleaner 3 3 3 3 3 <none> 68m
NAME READY STATUS RESTARTS AGE
pod/file-cleaner-6bn4t 1/1 Running 0 12m
pod/file-cleaner-gpntb 1/1 Running 0 12m
pod/file-cleaner-j8tf5 1/1 Running 0 11m
pod/web-app-75f6bd9b69-fnfkw 1/1 Running 0 122m
pod/web-app-75f6bd9b69-lk24l 1/1 Running 0 122m
pod/web-app-75f6bd9b69-wsnfd 1/1 Running 0 122m
# Delete both the Deployment and the DaemonSet
$ kubectl delete deployment.apps/web-app daemonset.apps/file-cleaner
$ kubectl get deploy,ds,pods
No resources found in default namespace.Conclusion
In this post, I performed an end-to-end demonstration of creating a file clean-up utility DaemonSet running on each Kubernetes cluster node, performing a rolling update and rollback on a DaemonSet.
This task utilized the previously learned Deployment and Persistent Storage Management skills as part of the Mastering Kubernetes One Task at a Time series.
Thank you for following and staying in touch. I look forward to seeing you soon with yet another new task in the series.
If you benefited from reading the post, please 👏 a few times before parting, and help others by sharing it; I highly appreciate that!
Please follow to stay in touch, track, and be the first to get notified of all future writings on AWS Cloud, Containers, Kubernetes, and Machine Learning. Also, check all my stories on The AWS Way publication.
