Summary

The article provides a guide on deploying a Ceph Quincy cluster using Ubuntu 22.04 with multiple monitor and OSD nodes, incorporating host-labels for flexible daemon placement.

Abstract

The article outlines the process for deploying a Ceph storage cluster using the Quincy release on Ubuntu 22.04 servers. It builds upon a previous guide that covered a single monitor node setup, expanding to a multi-monitor configuration for enhanced redundancy and high availability. The deployment utilizes Docker as the container runtime and emphasizes the use of host-labels for organizing and managing the placement of Ceph daemons across monitor and OSD nodes, which allows for better adaptability in the event of host failures or maintenance. The article also addresses potential issues during the installation process, such as repository errors, and provides solutions for them.

Opinions

The author considers password-less SSH configuration unnecessary for the deployment user, which may diverge from common practice in automated deployments.
Time synchronization across nodes is highlighted as a necessary step for a successful Ceph deployment.
The use of Ceph's built-in orchestrator module for managing host labels and daemon placement is favored for its flexibility in managing the cluster.
The author suggests that the cephadm tool is optionally installed from the repository, but the downloaded executable is sufficient for deployment, indicating a preference for manual installation in certain scenarios.
The article implies that the automatic creation of the _admin label and its associated configuration files on the bootstrap node simplifies cluster access and management for administrators.

Deployment of Multi-Monitor Ceph

With Quincy, Ubuntu 22.04 and host-labels

In a previous article, we described how to deploy a Ceph cluster with one monitor node and two OSD nodes. We used Ubuntu 20.04 and docker.io package and installed the Ceph pacific version.

Here, we will deploy Ceph quincy version with multiple monitors and OSD nodes. We will use Ubuntu 22.04 and docker.io.

In this article we assume you are familiar to the basic deployment steps in the previous article.

First we need to prepare the nodes for deployment: a user with sudo permission, dns entries (optional, but preferable), and container runtime (docker or podman). Please note that password-less ssh configuration for this user is unnecessary. Time-sync among nodes are also necessary.

Now, let’s start with getting a cephadm for quincy.

curl --silent --remote-name --location https://github.com/ceph/ceph/raw/quincy/src/cephadm/cephadm
sudo ./cephadm add-repo --release quincy # Install cephadm

As of 2023/06/27, installing cephadm will produce this error.

The repository 'https://download.ceph.com/debian-quincy jammy Release' does not have a Release file.

If this happens, we can remove /etc/apt/sources.list.d/ceph.list and run sudo apt update. Please note that installing cephadm is optional and the downloaded executable is good enough for deployment.

Here, we have three monitor nodes: ceph-mon1, ceph-mon2 and ceph-mon3. We have 4 OSD nodes: ceph-osd01, ceph-osd02, ceph-osd03 and ceph-osd04. This time we will use labels for the deployment.

sudo ceph orch host label add ceph-mon1 mon
sudo ceph orch host label add ceph-mon2 mon
sudo ceph orch host label add ceph-mon3 mon
sudo ceph orch host label add ceph-osd01 osd
sudo ceph orch host label add ceph-osd02 osd
sudo ceph orch host label add ceph-osd03 osd
sudo ceph orch host label add ceph-osd04 osd

After these commands, we will have these hosts with labels.

To deploy more monitors on the mon nodes, run this command.

sudo ceph orch apply mon --placement="3 label:mon"

The placement with labels can be applied for any daemons.

sudo ceph orch apply prometheus --placement 'count:1 label:mon'
sudo ceph orch apply alertmanager --placement 'count:1 label:mon'
sudo ceph orch apply grafana --placement 'count:1 label:mon'

sudo ceph orch apply mgr --placement="2 ceph-mon1 ceph-mon2" # hostname can be used as well.

After these configurations are applied, we have the following. The exact placement is left to the Ceph cluster. As you see, the most daemons are on ceph-mon1. The output were trimmed for conciseness.

It is worthwhile to pay attention on _admin label. When we bootstrap ceph-mon1 node, the label was created automatically and it is a special label. Once this label is applied to a node, two files will be created in /etc/ceph directory: ceph.conf and ceph.client.admin.keyring. Once the ceph-common package is installed, this can access the Ceph cluster.

In this article, we explained how to deploy a Ceph cluster where

Ceph version is Quincy
The OS is Ubuntu 22.04 and the container runtime is docker.io
host labels are used for daemon placement. This placement will give freedom to move daemon in case some hosts are down.