Automation Suite
2023.10
false
Failure to upload or download data in objectstore - Automation Suite 2023.10
Banner background image
logo
Automation Suite on Linux Installation Guide
Last updated Feb 13, 2024

Failure to upload or download data in objectstore



Description

This issue may occur when the object-store state is in a degraded state due to a placement group (PG) inconsistency.

Verify if the problem is indeed related to rook-ceph PG inconsistency by running the following commands:

export KUBECONFIG=/etc/rancher/rke2/rke2.yaml PATH=$PATH:/var/lib/rancher/rke2/bin
ROOK_CEPH_TOOLS=$(kubectl -n rook-ceph get pods | grep rook-ceph-tools)
kubectl -n rook-ceph exec -it $ROOK_CEPH_TOOLS -- ceph statusexport KUBECONFIG=/etc/rancher/rke2/rke2.yaml PATH=$PATH:/var/lib/rancher/rke2/bin
ROOK_CEPH_TOOLS=$(kubectl -n rook-ceph get pods | grep rook-ceph-tools)
kubectl -n rook-ceph exec -it $ROOK_CEPH_TOOLS -- ceph status

If the problem is related to a rook-ceph PG inconsistency, the output will contain the following messages:



....
....
Possible data damage: X pgs inconsistent
....
....
X active+clean+inconsistent
....
........
....
Possible data damage: X pgs inconsistent
....
....
X active+clean+inconsistent
....
....

Solution

To repair the inconsistent PG, take the following steps:

  1. Exec to rook-ceph tools:

    kubectl -n rook-ceph exec -it $ROOK_CEPH_TOOLS -- shkubectl -n rook-ceph exec -it $ROOK_CEPH_TOOLS -- sh
  2. Trigger the rook-ceph garbage collector process. Wait until the process is complete.

    radosgw-admin gc processradosgw-admin gc process
  3. Find a list of active+clean+inconsistent PGs:
    ceph health detail
    # output of this command be like
    # ....
    # pg <pg-id> is active+clean+inconsistent, acting ..
    # pg <pg-id> is active+clean+inconsistent, acting ..
    # ....
    #ceph health detail
    # output of this command be like
    # ....
    # pg <pg-id> is active+clean+inconsistent, acting ..
    # pg <pg-id> is active+clean+inconsistent, acting ..
    # ....
    #
  4. Trigger a deep scrub on the PGs one at a time. This command takes few minutes to run, depending on the PG size.

    ceph pg deep-scrub <pg-id>ceph pg deep-scrub <pg-id>
  5. Watch the scrubbing status:

    ceph -w | grep <pg-id>ceph -w | grep <pg-id>
  6. Check the PG scrub status. If the PG scrub is successful, the PG status should be active+clean+inconsistent.
    ceph health detail | grep <pg-id>ceph health detail | grep <pg-id>
  7. Repair the PG:

    ceph pg repair <pg-id>ceph pg repair <pg-id>
  8. Check the PG repair status. The PG ID should be removed from the active+clean+inconsistent list if the PG is repaired successfully.
    ceph health detail | grep <pg-id>ceph health detail | grep <pg-id>
  9. Repeat steps 3 to 8 for the rest of the inconsistent PG.
  • Description
  • Solution
Support and Services icon
Get The Help You Need
UiPath Academy icon
Learning RPA - Automation Courses
UiPath Forum icon
UiPath Community Forum
Uipath Logo White
Trust and Security
© 2005-2024 UiPath. All rights reserved.