All Longhorn replicas are faulted

Description

If the Longhorn replicas are in a faulted state and require manual salvage, the volume may not be able to attach and remain in a detach state.

To check if the volume requires manual salvage, run the following commands:

kubectl  logs -l app=longhorn-manager  -n longhorn-system  -c longhorn-manager --prefix=true  --tail=-1  |grep "set engine salvageRequested to true" | grep <PV NAME>

Sample output:

2023-11-20T18:22:16.667609096+11:00 time="2023-11-20T07:22:16Z" level=info msg="All replicas are failed, set engine salvageRequested to true" accessMode=rwo controller=longhorn-volume frontend=blockdev migratable=false node=rpa-suite-dev-01.it.csiro.au owner=rpa-suite-dev-01.it.csiro.au state=detaching volume=pvc-031fd6bc-9cfe-420a-9213-da38509d733a

Solution

To fix this issue, take the following steps:

1. Scale down the workload pods.

2. Find replicas of the relevant volume by running the following command:

   kubectl get replicas.longhorn.io -n longhorn-system |grep <PV_NAME>

3. Edit the replicas.longhorn.io object for the relevant PV by running the following command and set the spec.failedat field to empty(““).

   kubectl get replicas.longhorn.io -n longhorn-system |grep <PV_NAME>

4. Scale up the workload pods.