RabbitMQ pod stuck in CrashLoopBackOff

Description

This issue causes the RabbitMQ pod to be stuck in CrashLoopBackOff, with the log of the failing pod showing wal_checksum_validation_failure as a reason.

To get a list of all pods, run the following command:

kubectl -n rabbitmq get pods

To get the logs of a pod, run the following command:

kubectl -n rabbitmq logs <CrashLoopBackOff-Pod-Name>

Solution

To fix the issue, take the following steps:

1. Find the number of RabbitMQ replicas:

   rabbitmqReplicas=$(kubectl -n rabbitmq get rabbitmqcluster rabbitmq -o json | jq -r '.spec.replicas')

2. Scale down the RabbitMQ replicas:

   kubectl -n rabbitmq patch rabbitmqcluster rabbitmq -p "{\"spec\":{\"replicas\": 0}}" --type=merge

   kubectl -n rabbitmq scale sts rabbitmq-server --replicas=0

3. Wait until all RabbitMQ pods are terminated:

   kubectl -n rabbitmq get pod

4. Check for pending volume snapshots:

   kubectl get volumesnapshot -n rabbitmq  <crashloopbackupoff_pod_pvc_name> | grep false

5. If any pending volume snapshot exists, delete it:

   kubectl delete volumesnapshot -n rabbitmq <pending_volume_snapshot_name>

6. Find and delete the PVC of the RabbitMQ pod that is stuck in the CrashLoopBackOff state:

   kubectl -n rabbitmq get pvc

   kubectl -n rabbitmq delete pvc <crashloopbackupoff_pod_pvc_name>

7. Scale up the RabbitMQ replicas:

   kubectl -n rabbitmq patch rabbitmqcluster rabbitmq -p "{\"spec\":{\"replicas\": $rabbitmqReplicas}}" --type=merge

8. Check if all RabbitMQ pods are healthy:

   kubectl -n rabbitmq get pod