手册：将 Ceph 数据池从复制类型迁移到纠删码类型

步骤 1：选择 Ceph 对象的路径

您必须选择具有可用于保存 Ceph 对象的存储空间的文件系统路径。 例如，假设您决定在server0 Kubernetes 节点上使用/ceph-data路径。

重要提示： 您必须对齐 Ceph 工具才能使用此主机路径，并且必须在同一台计算机上执行所有后续命令（在本例中为 server0）。

export ROOK_CEPH_EXPORT_PATH="/ceph-data"

要查看 Ceph 集群中的对象使用的存储空间，请运行以下命令：

ceph_objects_bytes=$(kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph status --format json | jq -r '.pgmap.data_bytes')
numfmt --to=iec-i $ceph_objects_bytes

步骤 2：准备 Ceph 工具以使用该路径

要准备 Ceph 工具以使用在步骤 1 中选择的路径，请执行以下步骤：

1. 禁用rook-ceph-object-store的自我修复:

   kubectl -n argocd patch application rook-ceph-operator --type=json -p '[{"op":"replace","path":"/spec/syncPolicy/automated/selfHeal","value":false}]'
   kubectl -n argocd patch application rook-ceph-object-store --type=json -p '[{"op":"replace","path":"/spec/syncPolicy/automated/selfHeal","value":false}]'
   kubectl -n argocd patch application fabric-installer --type=json -p '[{"op":"replace","path":"/spec/syncPolicy/automated/selfHeal","value":false}]'

2. 编辑 Ceph 工具部署以在 Kubernetes 节点 server0 上装载${ROOK_CEPH_EXPORT_PATH}

   kubectl -n rook-ceph patch deploy rook-ceph-tools --type='json' -p='[{"op": "add", "path":"/spec/template/spec/nodeName", "value": "server0"},{"op": "add", "path":"/spec/template/spec/volumes/2", "value": {"name":"ceph-export", "hostPath": {"path": "'${ROOK_CEPH_EXPORT_PATH}'", "type":"Directory"}  }}, {"op":"add", "path": "/spec/template/spec/containers/0/volumeMounts/2", "value": {"name": "ceph-export", "mountPath": "'${ROOK_CEPH_EXPORT_PATH}'"}},{"op": "remove", "path": "/spec/template/spec/containers/0/resources/limits"}]'
   kubectl -n rook-ceph rollout status deploy rook-ceph-tools

3. 允许工具 pod 在${ROOK_CEPH_EXPORT_PATH}内写入

   chmod 777 ${ROOK_CEPH_EXPORT_PATH}

步骤 3：阻止其他命名空间对rook-ceph命名空间的访问

1. 阻止从rook-ceph命名空间以外的任何其他命名空间流向rook-ceph命名空间的流量：

   kubectl apply -f - <<EOF
   kind: NetworkPolicy
   apiVersion: networking.k8s.io/v1
   metadata:
     namespace: rook-ceph
     name: block-rook-ceph-from-other-ns
   spec:
     podSelector:
       matchLabels:
     ingress:
     - from:
       - podSelector: {}
   EOF

2. 重新启动 RGW 部署，以关闭已与其他命名空间建立的连接：

   for rgw_deploy in $(kubectl -n rook-ceph get deploy -l app=rook-ceph-rgw  -o name);do
       kubectl -n rook-ceph rollout restart "${rgw_deploy}"
       kubectl -n rook-ceph rollout status "${rgw_deploy}"
   done

步骤 4：查看集群对象计数

要查看集群对象计数，请运行以下命令：

BEFORE_MIGRATION_DATA_POOL_OBJECT_COUNT=$(kubectl -n rook-ceph exec deploy/rook-ceph-tools -- rados df --format json | jq -r --arg poolName "rook-ceph.rgw.buckets.data" '.pools[] | select(.name==$poolName).num_objects')
echo "BEFORE_MIGRATION_DATA_POOL_OBJECT_COUNT=${BEFORE_MIGRATION_DATA_POOL_OBJECT_COUNT}"

建议在迁移后重新检查对象计数，以确保没有发生数据丢失。

步骤 5：导出 Ceph 数据池

要导出 Ceph 数据池，请运行以下命令：

nohup kubectl -n rook-ceph exec deploy/rook-ceph-tools -- rados -p 'rook-ceph.rgw.buckets.data' export --workers 5   ${ROOK_CEPH_EXPORT_PATH}/ceph-data-pool >> /tmp/ceph-data-pool-export.log 2>&1 & 
wait $!
if [[ $? -eq 0 && -f ${ROOK_CEPH_EXPORT_PATH}/ceph-data-pool ]]; then
  echo "Export ran successfully"
else
 echo "Error while running export"
fi

步骤 6：缩小rook-ceph运算符

要缩减rook-ceph运算符，请运行以下命令：

kubectl -n rook-ceph scale --replicas=0 deployment/rook-ceph-operator

步骤 7：重新创建纠删码池

要重新创建纠删码池，请运行以下命令：

kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph osd pool rm rook-ceph.rgw.buckets.data rook-ceph.rgw.buckets.data --yes-i-really-really-mean-it --yes-i-really-really-mean-it-not-faking
kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph osd crush rule rm rook-ceph.rgw.buckets.data
kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph osd erasure-code-profile set rook-ceph_ecprofile k=2 m=1  crush-failure-domain=host
kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph osd pool create rook-ceph.rgw.buckets.data erasure rook-ceph_ecprofile
kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph osd pool set rook-ceph.rgw.buckets.data compression_mode none
kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph osd pool application enable rook-ceph.rgw.buckets.data rook-ceph-rgw  --yes-i-really-mean-it

步骤 8：将数据导入数据池

要将数据导入数据池，请运行以下命令：

nohup kubectl -n rook-ceph exec deploy/rook-ceph-tools -- rados -p 'rook-ceph.rgw.buckets.data'  import --workers 5   ${ROOK_CEPH_EXPORT_PATH}/ceph-data-pool >> /tmp/ceph-data-pool-import.log 2>&1 & 
wait $!
if [[ $? -eq 0 ]]; then
  echo "Import ran successfully"
else
 echo "Error while running import"
fi

步骤 9：验证加载的数据

要验证加载的数据，请运行以下命令：

try=120
return_code=1
for index in $(seq 0 "${try}"); do
  AFTER_MIGRATION_DATA_POOL_OBJECT_COUNT=$(kubectl -n rook-ceph exec deploy/rook-ceph-tools -- rados df --format json | jq -r --arg poolName "rook-ceph.rgw.buckets.data" '.pools[] | select(.name==$poolName).num_objects')
  if [[ $AFTER_MIGRATION_DATA_POOL_OBJECT_COUNT -eq $BEFORE_MIGRATION_DATA_POOL_OBJECT_COUNT ]]; then
    return_code=0
    break
  fi
  [[ $index -eq $try ]] || sleep 5
done
if [[ $return_code -eq 0 ]]; then
  echo "Found equal object count(${BEFORE_MIGRATION_DATA_POOL_OBJECT_COUNT})"
else
  echo "Found difference in object count for pool before(${BEFORE_MIGRATION_DATA_POOL_OBJECT_COUNT}) and after(${AFTER_MIGRATION_DATA_POOL_OBJECT_COUNT})"
  echo "Please raise a support ticket with uipath to complete the migration"
fi

步骤 10：还原临时更改

要还原临时更改，请运行以下命令：

kubectl -n argocd patch application rook-ceph-operator --type=json -p '[{"op":"replace","path":"/spec/syncPolicy/automated/selfHeal","value":true}]'
kubectl -n argocd patch application rook-ceph-object-store --type=json -p '[{"op":"replace","path":"/spec/syncPolicy/automated/selfHeal","value":true}]'
kubectl -n argocd patch application fabric-installer --type=json -p '[{"op":"replace","path":"/spec/syncPolicy/automated/selfHeal","value":true}]'
kubectl -n rook-ceph scale --replicas=1 deployment/rook-ceph-operator
kubectl -n rook-ceph patch deploy rook-ceph-tools --type='json' -p='[{"op": "remove", "path":"/spec/template/spec/nodeName"},{"op": "remove", "path":"/spec/template/spec/volumes/2"}, {"op":"remove", "path": "/spec/template/spec/containers/0/volumeMounts/2"},{"op": "add", "path": "/spec/template/spec/containers/0/resources/limits", "value": {"memory": "256Mi"}}]'
kubectl -n rook-ceph delete NetworkPolicy block-rook-ceph-from-other-ns

步骤 11：更新 ArgoCD 配置

现在，您必须确保使配置和实际状态保持同步。 为此，请运行以下命令来更新 ArgoCD 配置：

kubectl  -n argocd get application  fabric-installer -o json | jq 'if ([.spec.source.helm.parameters[].name] | index ("global.rook.dataPoolType")) == null then .spec.source.helm.parameters +=  [{"name": "global.rook.dataPoolType" , "value": "erasure-coded"}] else (.spec.source.helm.parameters[] | select(.name == "global.rook.dataPoolType").value) |= "erasure-coded" end'  | kubectl apply  -f -