Summary

The website content provides a step-by-step guide on setting up a virsh pool using a Ceph RBD pool for virtual machine images, including creating a RBD pool, a Ceph user, installing necessary packages, and defining a virsh pool with secrets.

Abstract

The article outlines the process of configuring Ceph RBD (RADOS Block Device) for storing virtual machine images. It begins with the creation of a dedicated RBD pool within Ceph, followed by the establishment of a specific user with permissions limited to this pool to enhance security. The guide then transitions to the client setup, emphasizing the installation of essential packages such as ceph-common and libvirt-daemon-driver-storage-rbd. It also clarifies that the ceph-common package may not be necessary, and a restart of the libvirtd service is required after installing the RBD storage driver. The final steps involve creating a virsh secret using a UUID and the key of the Ceph user, and defining and starting the virsh pool using an XML configuration file. The article concludes by highlighting the advantages of using Ceph RBD for VM images, such as the ability to leverage Ceph's features for improved performance and scalability.

Opinions

The article assumes that the reader has a pre-existing KVM/libvirt environment and does not cover the setup of this environment.
It is recommended not to use Ceph admin credentials for the RBD pool operations to maintain security best practices.
The article provides troubleshooting tips, such as handling errors related to missing RBD backend packages and restarting the libvirtd service if necessary.
The author acknowledges that there may be warnings when using credential information in plain text but suggests ignoring them for simplicity, with a note on the importance of securely handling credentials in a production environment.
The use of Ceph RBD for VM images is presented as beneficial due to Ceph's inherent features, such as data redundancy, high availability, and dynamic resizing of images.

Ceph RBD for Virtual Machine Images

How to set up a virsh pool backed by a RDB pool

One of most commonly used cases for Ceph rbd images is for virtual machine images. Here we will go through a sequence of steps to set up a virsh pool using a rbd pool. The overview of these steps are as follows.

Create a rbd pool
Create a user for the rbd pool
Client Setup, Part 1: Install packages
Client Setup, Part 2: Define a virsh pool

Please note that we assume that kvm package and environment is already set. We do not address how to set up kvm/libvirt environments.

Create a rbd pool

We covered this step in the Ceph RBD Image mount article. We chose libvirt-pool as a rbd pool name.

sudo ceph osd pool create libvirt-pool
sudo rbd pool init libvirt-pool

These commands need to be run on the node where ceph admin credentials are available, usually, one of ceph monitor nodes.

Create a Ceph user for the rbd pool

All the nodes that will use this pool, nodes where virtual machines are running, need a Ceph credential to operate this pool. So we do not want to use Ceph admin credenitals. We will create a Ceph client for this pool only.

sudo ceph auth get-or-create client.libvirt mon 'profile rbd' osd 'profile rbd pool=libvirt-pool' mgr 'profile rbd pool=libvirt-pool'

Here, we just provide a command for user creation. Ceph user creation is previously covered here. We did not save keyring file this time though. As you will see, we do not directly use its keyring file.

Client Setup, Part 1: Install packages

As usual, we need to install ceph-common package. The second package libvirt-daemon-driver-storage-rbd is necessary, but sometimes I had to install explicitly the other times it was automatically installed.

sudo apt install ceph-common
sudo apt install libvirt-daemon-driver-storage-rbd

If the package is not installed, you will get the following error along the way.

error: Failed to define pool from libvirt-rbd-pool.xml
error: internal error: missing backend for pool type 9 (rbd)

[Edit] : ceph-common package is not required. libvirtd.service need to be restarted after libvirt-daemon-driver-storage-rbd.

Client Setup, Part 2: Define a virsh pool

We will create a virsh secret. First, generate a UUID using uuidgen. Then, create a xml file for a virsh secret.

Please note that 1f7a22e9-d181–445a-ab27–16f59c2ad184 is generated by uuidgen. Now, create a virsh secret and check it with the following commands.

virsh secret-define --file libvirt-secret.xml
virsh secret-list

Its last step is to set the value of the secret. The value should be the key of the ceph user we created in the first step. To get it, use this command on a ceph monintor node.

sudo ceph auth get-key client.libvirt

Then, we have all the values: uuid and the client key.

virsh secret-set-value --secret "${UUID}" --base64 "${CLIENT_KEY}"

There will be some warnings from using credential information in a plain text, which I happily ignored. Otherwise, I need to copy admin credentials first and generate the client key in the command itself. Then, for security concerns, probably I would like to delete it later.

Now, we need to define a pool. The first step is to create a xml file that describe pool.

This should be self-explanatory. The ceph cluster we have been using for this series of articles are described in here. The host in the xml file is the ceph monitor node. If there are multiple monitor nodes, it is better to include all of them.

Now, we run the following commands to set up a virsh pool.

virsh pool-define libvirt-rbd-pool.xml
virsh pool-list --all                      # To check
virsh pool-autostart libvirt-pool
virsh pool-start  libvirt-pool

When you run the first command, you may encounter the error described in the install package step. Another path you may encounter is that libvirtd service did not load it. If the package is already installed but the error occurs, you may want to restart libvirtd.

sudo systemctl restart libvirtd

With rbd image features, hosting VM images on a Ceph cluster has some benefits. An example of creating a vm image is shown below. Enjoy

virsh vol-create-as "libvirt-pool" "[vm-name]" --capacity "128G" --format raw