Unattended Setup of Single Node OpenShift Cluster
Single node OpenShift cluster is ideal for edge computing or POC projects due to the scaled down resource requirement. Additionally, provision a cluster in a fully automated unattended way is the first important step towards a successful POC. In this paper, let’s explore the unattended approach to setup a single node OpenShift cluster.
Bastion Node
I am using KVM to provision VMs for the OCP cluster.
The bastion node to access the cluster is required. I am using a Ubuntu node for the bastion node.
As we have one single node cluster, the LB to route the traffic for the cluster API or the application services is not required.
The DNS to serve the A record and PTR record for the cluster during the bootup and runtime later is still required.
We will setup the DNS server on the bastion node.
Install the dnsmasq package with the apt command, then configure the dnsmasq with the following configuration files under /etc/dnsmasq.d
common.conf
# forward, use original DNS server
server=10.0.xxx.yyy
server=10.0.aaa.bbbocp413–1node.conf
address=/ocp413-1node-bastion.ocp413-1node.ibmcloud.io.cpak/192.168.10.209
ptr-record=209.10.168.192.in-addr.arpa,ocp413-1node-bastion.ocp413-1node.ibmcloud.io.cpak
address=/ocp413-1node-node.ocp413-1node.ibmcloud.io.cpak/192.168.10.201
ptr-record=201.10.168.192.in-addr.arpa,ocp413-1node-node.ocp413-1node.ibmcloud.io.cpak
address=/api.ocp413-1node.ibmcloud.io.cpak/192.168.10.201
address=/api-int.ocp413-1node.ibmcloud.io.cpak/192.168.10.201
address=/.apps.ocp413-1node.ibmcloud.io.cpak/192.168.10.201The conf file above defines both the A record and the PTR record for the master node. It also define the api and api-int for the cluster API and the wildcard domain name for the applications.
Update the systemd-resolve service configuration, /etc/systemd/resolved.conf
[Resolve]
DNS=127.0.0.1
DNSStubListener=noRestart the systemd-resolve and the dnsmsq service. Now we have the cluster DNS service available at the bastion node.
Download the tools
The official approach of installing OCP is to bootup the node with the RHCOS iso file, interactively setup the network, and install the coreos with by running the coreos-installer command interactively. For the single node cluster, the node first boot up as the bootstrap node, and then reboot as the master node.
Let’s automate this process. With a single ISO file, the master node will be setup with all the required parameters embeded in the ISO file.
Install the toolings such as podman, oc, openshift-install. Extract the coreos-installer comand from the latest coreos-installer image.
podman run --rm -v $(pwd):/data \
--entrypoint bash \
quay.io/coreos/coreos-installer:release \
-c "cp /sbin/coreos-installer /data"Download the matched ISO file using the openshift-install command,
export ISO_URL=$(openshift-install coreos print-stream-json | grep location | grep x86_64 | grep iso | cut -d\" -f4)
echo downloading $ISO_URL
axel -o rhcos-live.iso --quiet $ISO_URL Customise the ISO file
Create the ocp install-config.yaml file as below,
apiVersion: v1
baseDomain: ibmcloud.io.cpak
compute:
- name: worker
replicas: 0
controlPlane:
name: master
replicas: 1
metadata:
name: ocp413-1node
networking:
clusterNetwork:
- cidr: 10.128.0.0/14
hostPrefix: 23
machineNetwork:
- cidr: 10.0.0.0/16
networkType: OVNKubernetes
serviceNetwork:
- 172.30.0.0/16
platform:
none: {}
bootstrapInPlace:
installationDisk: /dev/vda
pullSecret: '{
"auths": {
"cloud.openshift.com": {
...
}'
sshKey: |
ssh-rsa ...
The bootstrapInPlace disk is where the installer install the master coreos after its first boot up as the bootstrap node.
Create the single node ignition config,
mkdir -p ocp
cp install-config.yaml ocp/
openshift-install --dir=ocp create single-node-ignition-configThe ignition file is created as ocp/bootstrap-in-place-for-live-iso.ign Embed this file into the ISO image with coreos-installer command by creating a new ISO file from the original one.
coreos-installer iso ignition embed \
-fi ocp/bootstrap-in-place-for-live-iso.ign
-o ocp413-1node-node.iso
rhcos-live.isoLets automate the network setup by updating the kernel argument,
coreos-installer iso kargs modify -a \
'ip=192.168.10.201::192.168.10.1::ocp413-1node-node.ocp413-1node.ibmcloud.io.cpak:ens3:none:192.168.10.209'The ip kernel parameter is using the format as of below,
ip={{ .ip }}::{{ .gateway }}:{{ .mask }}:{{ .hostname }}:{{ .iface }}:none:{{ .nameserver }}You can check the kernel parameter and the embedded ignition file with
coreos-installer iso kargs show ocp413-1node-node.iso
coreos-installer iso ignition show ocp413-1node-node.isoProvision the Node and Get Cluster Ready
With KVM, we can now create the VM using the ISO created above,
qemu-img create -f qcow2 /data1/kvm-images/ocp413-1node-node.qcow2 200G && virt-install --name=ocp413-1node-node --ram=16384 --vcpus=8 --disk path=/data1/kvm-images/ocp413-1node-node.qcow2,bus=virtio,cache=none --noautoconsole --graphics=vnc --network network=ocp,model=virtio --boot hd,cdrom --cdrom /ocpIso/ocp413-1node-node.iso'Watch the VM started and shutdown after the boot up. Bring it up with virsh start ocp413–1node-node . The VM will then restart it self couple of times till the OCP master fully ready. It’s noted that after the 1st restart, the node serves as the bootstrap node, it calls the coreos-installer program to save as the master node copying the same network settings, and after restart this node is the master node.
We could automate this process with the following magefile task.
func (G03_node) T03_wait_to_restart() {
lo.AttemptWithDelay(30, 40*time.Second, func(index int, dur time.Duration) error {
cmd := quote.CmdTemplate(`
virsh list | grep {{ .name }}
`, map[string]string{
"name": config.String("ocpMasterNode.name"),
})
log.Printf("waiting for node %s to restart, i=%d, duration=%s", config.String("ocpMasterNode.name"), index, dur)
err := master.Execute(cmd)
if err != nil {
log.Printf("node %s is stopped", config.String("ocpMasterNode.name"))
return nil
} else {
return errors.New("node is not stopped")
}
})
cmd := quote.CmdTemplate(`
sleep 5
virsh start {{ .name }}
`, map[string]string{
"name": config.String("ocpMasterNode.name"),
})
master.Execute(cmd)
}Wait for the cluster to be ready by running,
openshift-install --dir=ocp wait-for install-completeNow a single node cluster is ready.
We can then configure the htpasswd for authentication purpose.
Adding Worker Nodes
The saved resource can be created as worker nodes for the single node (master node) cluster. Same as the normal OCP cluster, we bootup the worker node with the worker’s ignition file.
Certainly before the bootup, we need to update the A and PTR records in the DNS for the workers.
Customise the ISO file to make the bootup process unatteneded. Not like the master node, we have to differentiate the live ISO boot up and the disk based boot up. In the coreos-installer program this is achieved with the different arguments such as--dest-ignition vs --live-ignition and--dest-karg-append vs--live-karg-append.
The command we use to create an unattended installation ISO file is as below,
coreos-installer iso customize
--dest-ignition ocp/worker.ign \
--dest-device /dev/vda \
--dest-karg-append ip=192.168.10.202::192.168.10.1::ocp413-1node-worker1.ocp413-1node.ibmcloud.io.cpak:ens3:none:192.168.10.209 \
-o ocp413-1node-worker1 rhcos-live.isoNote all the parameters are using the dest prefix, which is for the disk based bootup. We save the OCP worker.ign into the target disk, and add the network settings for disk boot up. (The network settings can be applied to the ISO based bootup also with the argument oflive-karg-append, but its not really required)
With the ISO file created, create the VM with KVM virt-install command.
Once the coreos is installed, the VM is shutdown. Start the VM, wait for the machine config operator to setup the worker node with couple of restart. Then similarly as the normal OCP cluster, approve the CSR multiple times till the worker node is ready.
All these can be automated using the same magefile approach shown above.
