Moving Ceph out of read-only mode

Move Ceph out of read-only mode when AI Center disk reaches full capacity in Automation Suite.

If you installed AI Center and use Ceph storage, take the following steps to move Ceph out of read-only mode:

1. Check if Ceph is at full capacity:

   kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph status
   

   If Ceph is at full capacity, you must adjust the read-only threshold to start up the rgw gateways.

2. Scale down the ML skills:

   kubectl -n uipath scale deployment <skill> --replicas=0
   

3. Put the cluster in write mode by raising the full-ratio threshold. The default value is 0.95; if Ceph remains in read-only mode after this change, increase the value incrementally (for example, to 0.96):

   ceph osd set-full-ratio 0.95
   

4. Run Garbage Collection:

   kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- radosgw-admin gc process --include-all
   

5. When storage goes down, run the following commands:

   kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph status
   kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph df
   

6. Once storage is no longer full, reset the full-ratio threshold back to the default value:

   ceph osd set-full-ratio 0.95
   

At this point, the storage should be lower, and the cluster should be healthy.