Understanding Kubernetes Etcd Back&Restore — A Beginner’s Guide

Practical Demonstration on etcd backup and restore

Find Complete mind map of A Beginner’s Guide to Kubernetes

In our previous article, we discussed the vital topic of maintaining a Kubernetes cluster, emphasizing the significance of etcd backup and restore, as well as the importance of staying up-to-date with cluster upgrades.

As promised, in this article, we are going to take a deep dive into the topic of etcd backup and restore in Kubernetes.

We’ll walk you through the data stored in etcd, introduce you to the powerful etcdctl tool for creating snapshots, and show you how to restore your cluster from these backups. With these skills, you’ll be well-equipped to backup your Kubernetes environment confidently.

let’s get started!

Check out “Understanding Kubernetes — A Beginner’s Guide” for the comprehensive series🚀

Introduction to etcdctl

We’ll use the etcdctl tool, which we previously installed for interacting with etcd.

etcdctl is a command-line utility used to manage etcd clusters, enabling you to read, write, and modify data stored in the distributed key-value store.

To begin, let’s find the etcd pod in our Kubernetes cluster:

$ kubectl get pods -A | grep etcd

# Output like...
kube-system     etcd-k8s-master                             1/1     Running     21 (26m ago)   47d

Once we locate the etcd pod, we can access it using the exec command:

$ kubectl exec -it --namespace kube-system etcd-k8s-master -- sh
sh-5.1#

Inside the etcd pod, we can check the version of etcd installed:

sh-5.1# etcd --version

etcd Version: 3.5.3
Git SHA: 0452feec7
Go Version: go1.16.15
Go OS/Arch: linux/amd64

Additionally, we can also check the version of etcdctl:

$ etcdctl version

etcdctl allows us to interact with etcd data directly from the command line. Through this article, you'll see how powerful this tool is for creating and restoring etcd snapshots,

So, let's move on to the next section to learn more about the data stored in etcd!

Backup etcd

We’ll explore how to back up the data stored in etcd using the etcdctl command-line tool. It is crucial to create regular snapshots of etcd data to ensure data integrity and have the ability to recover from disasters or unexpected failures.

Where to find etcd data?

For clusters created with kubeadm, etcd runs as a single-node cluster inside a pod, and its data is stored in the /var/lib/etcd directory. This directory is mounted to the master node using the hostPath mount type, allowing access to the etcd data from the host.

Let’s take a look at how the etcd data is mounted on the master node:

$ kubectl get pod --namespace kube-system etcd-k8s-master -o jsonpath='{.spec.containers[0].volumeMounts}' | jq
[
  {
    "mountPath": "/var/lib/etcd",
    "name": "etcd-data"
  },
  {
    "mountPath": "/etc/kubernetes/pki/etcd",
    "name": "etcd-certs"
  }
]

As you can see, the etcd data is mounted at /var/lib/etcd on the master node.

Snapshotting etcd Data

To create a snapshot (backup) of the etcd data, we use the etcdctl snapshot save command:

$ ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  snapshot save /var/lib/dat-backup.db

This command creates a snapshot file named dat-backup.db, containing the etcd data. It is important to decide where to store the backup data. You can store it locally on the master node or transfer it to an external storage system, such as a remote server or cloud storage, for additional redundancy.

Verifying the Snapshot

After taking the snapshot, it’s essential to verify its integrity. We can do this using the etcdctl snapshot status command:

$ ETCDCTL_API=3 etcdctl --write-out=table \
  snapshot status /var/lib/dat-backup.db

The output should look like this:

+----------+----------+------------+------------+
|   HASH   | REVISION | TOTAL KEYS | TOTAL SIZE |
+----------+----------+------------+------------+
| 9b862d61 | 100      | 3118       | 123.7 KB   |
+----------+----------+------------+------------+

This table displays the hash of the snapshot, the revision number, total keys, and total size. If the snapshot is valid, you will see this information.

With the snapshot in hand, we can confidently perform cluster upgrades or restore the cluster to a previous state in the event of any issues.

Restore etcd from Snapshot

We learned how to take a snapshot of the etcd data. Now, let’s dive into the process of restoring the etcd data from that snapshot.

Restoring etcd from a snapshot is a critical procedure, especially in scenarios where data corruption or cluster failure occurs.

By having a recent and reliable snapshot, we can quickly recover the etcd data and bring our Kubernetes cluster back to a stable state.

Restoring Data step by step

To restore etcd data from a snapshot, we use the etcdctl snapshot restore command. Before running this command, we must ensure that etcd is stopped on the master node where we intend to restore the data. Let's walk through the steps to restore the etcd data:

1.Stop the etcd service on the master node:

$ sudo systemctl stop etcd

2.Run the etcdctl snapshot restore command:

$ ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  snapshot restore /var/lib/dat-backup.db

This command restores the etcd data from the dat-backup.db snapshot file.

3. Start the etcd service:

$ sudo systemctl start etcd

After completing these steps, etcd should be up and running with the restored data from the snapshot.

Complete Backup and Restore Cycle

Now that we have seen how to take a snapshot and restore the etcd data, let’s briefly summarize the complete backup and restore cycle:

1. Taking a Snapshot: Use the etcdctl snapshot save command to create a snapshot of the etcd data. Store the snapshot in a safe location, either on the master node or an external storage system.
2. Restoring from a Snapshot: In case of data corruption or cluster failure, stop the etcd service on the master node, use the etcdctl snapshot restore command to restore the data from the snapshot, and then start the etcd service again.

In the next section, we’ll explore another crucial aspect of maintaining a Kubernetes cluster: upgrading the cluster to the latest version. Stay tuned to learn more about cluster upgrades!

🔔 Stay tuned or subscribe to my series: “Understanding Kubernetes — A Beginner’s Guide” to explore everything about Kubernetes. 🚀

➕Join the Medium Membership Program to support my work and connect with other writers.

📝 Have questions or suggestions? Leave a comment or message me through Medium. Let’s connect!

Thank you for your support! 🌟