Downgrading Ceph from 16.2.6 to 15.2.9

Automation Suite bundles Ceph objectstore within the cluster to store files such as NuGet feeds in Orchestrator, Datasets in AI Center, and more.

Automation Suite 2021.10.3 and 2021.10.4 ship with Ceph objectstore 16.2.6. This Ceph version introduces an issue that can potentially corrupt data. As a result, Ceph has officially recalled version 16.2.6 until the issue is resolved in a newer version.

Before upgrading to a newer Automation Suite version, you must ensure the Ceph issue does not affect your cluster and the data it stores. To do that, it is required to downgrade from Ceph 16.2.6 to 15.2.9, which is a stable version.

Downgrading Ceph from 16.2.6 to 15.2.9 directly is not possible, and there are a series of operations we carry out for you when upgrading to a new Automation Suite version:

1. Back up the Ceph data;
2. Destroy Ceph cluster version 16.2.6;
3. Recreate Ceph cluster version 15.2.9;
4. Upload all the backed-up Ceph data.

To back up the Ceph data before upgrading to a new Automation Suite version, you need a temporary folder. The folder must be big enough to store all the Ceph data. To determine the size of the temporary folder required for a Ceph data backup, run the following command:

ceph_object_size=$(kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph status --format json | jq -r '.pgmap.data_bytes')
echo "You need '$(numfmt --to=iec-i $ceph_object_size)' storage space"

Before the Automation Suite upgrade, you must also set the TMP_CEPH_BACKUP_PATH variable. To create a folder and set TMP_CEPH_BACKUP_PATH, run the following command:

# replace "/path/to/backup/ceph" with actual path where you will going to take backup
mkdir -p /path/to/backup/ceph
export TMP_CEPH_BACKUP_PATH="/path/to/backup/ceph"

Note: Make sure to replace /path/to/backup/ceph with the correct Ceph path in the previous command.

For additional steps, follow the upgrade documentation.