Pods stuck in Init state due to XFS volume mount failure on RHEL 8.x nodes

Resolve pods stuck in Init state due to a TopoLVM XFS formatting incompatibility with RHEL 8.x in Automation Suite.

Description

When a pod that requires a new persistent volume (for example, an AI Center training pipeline) starts on a RHEL 8.x node, the pod remains in the Init state with the following error:

Warning  FailedMount  36s (x11 over 8m50s)  kubelet  MountVolume.SetUp failed for volume "pvc-82575d2d-2269-4cef-ace8-fa1602f3dd44" : rpc error: code = Internal desc = mount failed: volume=6d81061d-79c4-46b3-9471-478c6c739b25, error=mount failed: exit status 32

Mounting command: mount
Mounting arguments: -t xfs -o nouuid /dev/uipathstorevg/6d81061d-79c4-46b3-9471-478c6c739b25 /var/lib/kubelet/pods/547bdf64-173b-4df1-ad7a-449f6796ca09/volumes/kubernetes.io~csi/pvc-82575d2d-2269-4cef-ace8-fa1602f3dd44/mount

Output: mount: /var/lib/kubelet/pods/547bdf64-173b-4df1-ad7a-449f6796ca09/volumes/kubernetes.io~csi/pvc-82575d2d-2269-4cef-ace8-fa1602f3dd44/mount: wrong fs type, bad option, bad superblock on /dev/mapper/uipathstorevg-6d81061d--79c4--46b3--9471--478c6c739b25, missing codepage or helper program, or other error.

The PVC is Bound, the PV looks healthy, and the underlying block device is visible with lsblk on the node.

Cause: TopoLVM formats new volumes with mkfs.xfs using on-disk features (bigtime, inobtcount, nrext64) introduced in newer xfsprogs/kernel versions. The kernel and xfsprogs shipped with RHEL 8.x can't mount an XFS volume created with these features, so the mount fails with wrong fs type, bad option, bad superblock.

This issue affects Automation Suite on Linux (RKE2) running on RHEL 8.x nodes only; RHEL 9.x and 10.x aren't affected.

Solution

To address the issue, patch the topolvm-node DaemonSet to format new volumes with a mkfs.xfs wrapper that disables the incompatible features:

1. Back up the current TopoLVM configuration:

   kubectl -n topolvm-system get ds topolvm-node -o yaml > /opt/UiPathAutomationSuite/topolvm-node-ds.yaml
   kubectl -n topolvm-system get all -o yaml > /opt/UiPathAutomationSuite/topolvm-system-all.yaml
   kubectl get storageclass local-volume-provisioner -o yaml > /opt/UiPathAutomationSuite/local-volume-provisioner-sc.yaml
   kubectl -n topolvm-system get configmap -o yaml > /opt/UiPathAutomationSuite/topolvm-system-configmaps.yaml
   

2. Create a wrapper ConfigMap that calls mkfs.xfs with RHEL 8.x-compatible options:

   cat <<'EOF' | kubectl apply -f -
   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: mkfs-xfs-wrapper
     namespace: topolvm-system
   data:
     mkfs.xfs: |
       #!/bin/sh
       exec /usr/sbin/mkfs.xfs -m crc=1,rmapbt=0,bigtime=0,inobtcount=0 -i nrext64=0 "$@"
   EOF
   

3. Verify the ConfigMap was created:

   kubectl -n topolvm-system get configmap mkfs-xfs-wrapper
   

4. Patch the topolvm-node DaemonSet to mount the wrapper over the default mkfs.xfs binary path:

   kubectl -n topolvm-system patch ds topolvm-node --type=strategic -p '
   spec:
     template:
       spec:
         containers:
         - name: topolvm-node
           volumeMounts:
           - mountPath: /usr/local/sbin/mkfs.xfs
             name: mkfs-xfs-wrapper
             readOnly: true
             subPath: mkfs.xfs
         volumes:
         - name: mkfs-xfs-wrapper
           configMap:
             name: mkfs-xfs-wrapper
             defaultMode: 493
             items:
             - key: mkfs.xfs
               mode: 493
               path: mkfs.xfs
   '
   

5. Roll out the DaemonSet and confirm the rollout completes:

   kubectl -n topolvm-system rollout status ds/topolvm-node