RabbitMQ pod stuck in CrashLoopBackOff

Description

This issue causes the RabbitMQ pod to be stuck in CrashLoopBackOff, with the log of the failing pod showing wal_checksum_validation_failure as a reason.

To get a list of all pods, run the following command:

kubectl -n rabbitmq get pods

To get the logs of a pod, run the following command:

kubectl -n rabbitmq logs <CrashLoopBackOff-Pod-Name>

Solution

To fix this issue, take the following steps:

1. Find the number of RabbitMQ replicas:

   rabbitmqReplicas=$(kubectl -n rabbitmq get rabbitmqcluster rabbitmq -o json | jq -r '.spec.replicas')

2. Scale down the RabbitMQ replicas:

   kubectl -n rabbitmq patch rabbitmqcluster rabbitmq -p "{\"spec\":{\"replicas\": 0}}" --type=merge

   kubectl -n rabbitmq scale sts rabbitmq-server --replicas=0

3. Wait until all RabbitMQ pods are terminated:

   kubectl -n rabbitmq get pod

4. Find and delete the PVC of the RabbitMQ pod that is stuck in CrashLoopBackOff state:

   kubectl -n rabbitmq get pvc

   kubectl -n rabbitmq delete pvc <crashloopbackupoff_pod_pvc_name>

5. Scale up the RabbitMQ replicas:

   kubectl -n rabbitmq patch rabbitmqcluster rabbitmq -p "{\"spec\":{\"replicas\": $rabbitmqReplicas}}" --type=merge

6. Check if all RabbitMQ pods are healthy:

   kubectl -n rabbitmq get pod