Summary

The web content provides a detailed guide on transferring virtual machine (VM) images from one Ceph cluster to another, including the necessary steps to update the VM's configuration for a new bare-metal machine.

Abstract

The article outlines a process for moving a VM image stored in a Ceph RBD pool from one Ceph cluster to another. It begins with obtaining the VM's domain configuration using virsh dumpxml. It then describes creating aliases for easier command execution and exporting the RBD image from the source cluster and importing it into the destination cluster. The guide includes updating the VM's domain XML configuration to reflect the new Ceph cluster's details, such as the secret UUID and monitor IP addresses. Finally, it covers defining the updated VM configuration on the new bare-metal host and starting the VM, as well as troubleshooting steps for potential errors related to outdated pool information.

Opinions

The author assumes familiarity with Ceph and libvirt, suggesting the guide is intended for an audience with experience in these technologies.
The use of aliases for command execution indicates a preference for efficiency and readability in command-line operations.
The article emphasizes the importance of updating the VM's configuration to match the new Ceph cluster's settings, highlighting the need for attention to detail in the migration process.
By providing troubleshooting advice, the author acknowledges common issues that may arise and offers practical solutions, showing a consideration for users who might encounter these problems.

Ceph: Moving Virtual Machine Image (Part I)

From a Ceph cluster to another Ceph cluster

This article covers moving VM images in a Ceph cluster as rbd image into another Ceph cluster. We also change the baremetal machine where VMs are running.

We assume that there are two Ceph clusters and each has libvirt-pool rbd pool and an associated client named libvirt.

The first step is to get domain.xml.

virsh dumpxml [vm]

The next step is to get image files from one Ceph cluster and put them into another cluster. Before we do that, we will create two aliases for conciseness.

alias rbd1="sudo rbd --conf ceph1.conf --id libvirt --keyring libvirt1.keyring"
alias rbd2="sudo rbd --conf ceph2.conf --id libvirt --keyring libvirt2.keyring"

Here, ceph1.conf and libvirt1.keyring are ceph.conf and libvirt.keyring for the first cluster. Similarly, ceph2.conf and libvirt2.keyring for the second cluster. This is an arbitrary choice. You can choose any file name.

For completeness, the commands that gives these two files are given below.

sudo ceph config generate-minimal-conf      # ceph.conf
sudo ceph auth get client.libvirt -o [file] # libvirt.keyring

Now, let’s get rbd images from Ceph cluster.

rbd1 export --pool libvirt-pool --image [rbd image] [filepath]

Then, we will put this image file into the other Ceph cluster.

rbd2 import --dest-pool libvirt-pool --dest [rbd image] [filepath]

This image move can happen at any node where both Ceph clusters are accessible. It does not need to be a bare-metal machine where the VM will run.

Now, we need to update domain xml file in the very first step. Here is a section where the storage image is defined. We need to change secret-uuid and monitor ip addresses. The example has three monitor nodes.

<disk type='network' device='disk'>
  <driver name='qemu' type='raw'/>
  <auth username='libvirt'>
    <secret type='ceph' uuid='secret-uuid'/>
  </auth>
  <source protocol='rbd' name='libvirt-pool/dk-podman-vm'>
    <host name='mon1.ip' port='6789'/>
    <host name='mon2.ip' port='6789'/>
    <host name='mon3.ip' port='6789'/>
  </source>
  <target dev='vda' bus='virtio'/>
  <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
</disk>

To see how to create a virsh secret, please see Ceph RBD for Virtual Machine Images article. We do not need to create a virsh pool for the new Ceph cluster. However, if you want to use this new pool, you may want to create and run a new virsh pool as well.

Now, we can define and start the VM on the new bare-metal node.

virsh define vm.xml
virsh start vm

Troubleshooting

When original rbd images are removed, the libvirtd may not have updated information and virsh command may trigger the following error.

failed to open the RBD image '[rbd image name]': No such file or directory

To fix this, refresh pool information with this command.

virsh pool-refresh [pool-name]