Cluster data appears lost after a reboot due to incorrect Rancher and etcd disk mount order

Restore hidden etcd data and correct the Rancher and etcd disk mount order after a reboot on Automation Suite on Linux.

Description

On multi-disk installations, the dedicated etcd disk, mounted at /var/lib/rancher/rke2/server/db, must be mounted after its parent directory, /var/lib/rancher. If the mount order is reversed, so that /var/lib/rancher gets mounted on top of an already-mounted etcd disk, the etcd data becomes hidden underneath the /var/lib/rancher mount.

A reboot that re-triggers this incorrect mount order makes the RKE2 server node appear to lose all persistent volumes, persistent volume claims, and applications, even though the underlying etcd data is still present on disk.

Starting with Automation Suite 2.2510.4, the uipathctl rke2 prereq run command checks for this condition. To run the check, use the following command:

./bin/uipathctl rke2 prereq run cluster_config.json --versions ./versions/helm-charts.json --log-level=debug --included disk --node-type server -v

If the mount order is incorrect, the check fails with the following error:

❌ [RKE2_DISK]
   ❌ [Rancher_Etcd_Mount_Order] '/var/lib/rancher/rke2/server/db' is not mounted. Configure the cluster disk (--cluster-disk-name) before continuing, or if both disks are already mounted, do the data migration if required, unmount and remount (or reboot) so '/var/lib/rancher' is mounted before '/var/lib/rancher/rke2/server/db'.

On earlier versions, or if you prefer not to run the full prereq check, verify the mount order directly:

stat -c '%n -> %D' /var/lib/rancher/ /var/lib/rancher/rke2/server/db /

If the device IDs are not all different, the etcd disk is not mounted at its expected path.

Solution

Important:

For a multi-node cluster, apply the following steps to one server node at a time, and confirm the node is healthy before moving to the next node. Back up the cluster and put it in maintenance mode before you begin. Run all commands as root (sudo -i).

To fix the mount order and restore the hidden etcd data, take the following steps:

1. Stop the node drain and RKE2 server services, then unmount both paths:

   systemctl stop node-drain.service
   systemctl stop rke2-server
   rke2-killall.sh
   umount /var/lib/rancher
   umount /var/lib/rancher/rke2/server/db
   

2. Mount the parent /var/lib/rancher volume only:

   RANCHER_UUID=$(blkid -o value -s UUID /dev/uipathvg/rancherlv)
   mount -t xfs -o defaults,nofail UUID="$RANCHER_UUID" /var/lib/rancher
   

3. Mount the etcd disk to a temporary location, so the hidden data is exposed instead of being covered again:

   mkdir -p /mnt/tmp_db
   ETCD_UUID=$(blkid -o value -s UUID /dev/uipathetcdvg/etcdlv)
   mount -t xfs -o defaults,nofail UUID="$ETCD_UUID" /mnt/tmp_db
   

4. Copy the hidden etcd data to the etcd disk:

   cp -ax /var/lib/rancher/rke2/server/db/. /mnt/tmp_db/
   

5. Unmount the temporary location, clear the stale data from the parent disk, and mount the etcd disk at its correct nested path:

   umount /mnt/tmp_db
   rmdir /mnt/tmp_db
   find /var/lib/rancher/rke2/server/db -mindepth 1 -delete
   mount -t xfs -o defaults,nofail UUID="$ETCD_UUID" /var/lib/rancher/rke2/server/db
   

6. Update /etc/fstab so the mount order persists across reboots. The parent mount (/var/lib/rancher) must be declared before the nested mount (/var/lib/rancher/rke2/server/db):

   # uipath-rancher
   UUID=<rancher-disk-uuid>    /var/lib/rancher    xfs    defaults,nofail    0    0
   # uipath-etcd
   UUID=<etcd-disk-uuid>    /var/lib/rancher/rke2/server/db    xfs    defaults,nofail    0    0
   

   Before rebooting, verify that both paths mount correctly:

   mountpoint /var/lib/rancher
   mountpoint /var/lib/rancher/rke2/server/db
   

7. Verify the mount order and restart the cluster services:

   df -h | grep rancher
   systemctl start rke2-server
   systemctl start node-uncordon