Automation Suite
2023.10
false
MongoDB pods in CrashLoopBackOff or pending PVC provisioning after deletion - Automation Suite 2023.10
Banner background image
logo
Automation Suite on Linux Installation Guide
Last updated Feb 13, 2024

MongoDB pods in CrashLoopBackOff or pending PVC provisioning after deletion

The MongoDB pods might get stuck in a cashback loop due to a corrupt PVC. The most probable cause of this issue is an unclean shutdown.

When experiencing this issue, the logs show the following:

Common point must be at least stable timestamp
{"t":{"$date":"2022-05-18T09:37:55.053+00:00"},"s":"W",  "c":"STORAGE",  "id":22271,   "ctx":"initandlisten","msg":"Detected unclean shutdown - Lock file is not empty","attr":{"lockFile":"/data/mongod.lock"}}
    ['currentState.Running' = false]
    ['currentState.IsVCRedistCorrect' = true]
    ['desiredState.ProcessType' != mongos ('desiredState.ProcessType' = mongod)]Common point must be at least stable timestamp
{"t":{"$date":"2022-05-18T09:37:55.053+00:00"},"s":"W",  "c":"STORAGE",  "id":22271,   "ctx":"initandlisten","msg":"Detected unclean shutdown - Lock file is not empty","attr":{"lockFile":"/data/mongod.lock"}}
    ['currentState.Running' = false]
    ['currentState.IsVCRedistCorrect' = true]
    ['desiredState.ProcessType' != mongos ('desiredState.ProcessType' = mongod)]

Recovery steps

  1. Delete the failing pod. If this solution does not work, continue to the next steps.

    kubectl delete pod <pod-name> -n mongodbkubectl delete pod <pod-name> -n mongodb
  2. Get the name of the corrupt PVC for the failing pods.
    kubectl -n mongodb get pvckubectl -n mongodb get pvc
  3. Delete the PVC for the failing pod.

    kubectl -n mongodb delete pvc <pvc-name>kubectl -n mongodb delete pvc <pvc-name>
    Note: At this point, the PVC should be auto-synced, and the pod should experience no issues anymore. If auto-provisioning does not happen, you need to perform the operation manually by taking the following steps.
  4. Get the PVC YAML for a healthy node.

    kubectl -n mongodb get pvc <pvc-name> -o yaml > pvc.yamlkubectl -n mongodb get pvc <pvc-name> -o yaml > pvc.yaml
  5. Edit the name and remove uuids/pvc-ids from the YAML.
  6. Remove the volume name and UID, and rename the PVC to the deleted PVC name.



  7. Apply the PVC.

    kubectl -n mongodb apply pvc.yamlkubectl -n mongodb apply pvc.yaml
  8. The PVC should be provisioned and attached to the PVC for the pod, and the pod should no longer experience any issues. If the pod does not resync, then delete it.
  • Recovery steps
Support and Services icon
Get The Help You Need
UiPath Academy icon
Learning RPA - Automation Courses
UiPath Forum icon
UiPath Community Forum
Uipath Logo White
Trust and Security
© 2005-2024 UiPath. All rights reserved.