Orchestrator Pod 处于 CrashLoopBackOff 状态或多次重新启动

描述

如果 Orchestrator Pod 处于 CrashLoopBackOff 状态或 1/2 多次重新启动后开始运行，则故障可能与对象存储提供程序 Ceph 的身份验证密钥有关。

要检查故障是否与 Ceph 相关，请运行以下命令：

kubectl -n uipath get pod -l app.kubernetes.io/component=orchestratorkubectl -n uipath get pod -l app.kubernetes.io/component=orchestrator

如果此命令的输出类似于以下选项之一，则需要运行其他命令。

Option 1:
NAME                            READY   STATUS    RESTARTS   AGE
orchestrator-6dc848b7d5-q5c2q   1/2     Running   2          6m1s
OR 
Option 2
NAME                            READY   STATUS             RESTARTS   AGE
orchestrator-6dc848b7d5-q5c2q   1/2     CrashLoopBackOff   6          16mOption 1:
NAME                            READY   STATUS    RESTARTS   AGE
orchestrator-6dc848b7d5-q5c2q   1/2     Running   2          6m1s
OR 
Option 2
NAME                            READY   STATUS             RESTARTS   AGE
orchestrator-6dc848b7d5-q5c2q   1/2     CrashLoopBackOff   6          16m

运行以下命令，验证失败是否与 Ceph 身份验证密钥有关：

kubectl -n uipath logs -l app.kubernetes.io/component=orchestrator | grep 'Error making request with Error Code InvalidAccessKeyId and Http Status Code Forbidden' -okubectl -n uipath logs -l app.kubernetes.io/component=orchestrator | grep 'Error making request with Error Code InvalidAccessKeyId and Http Status Code Forbidden' -o

如果上述命令的输出中包含字符串 Error making request with Error Code InvalidAccessKeyId and Http Status Code Forbidden，则失败的原因在于 Ceph 身份验证密钥。

解决方案

使用以下命令重新运行 rook-ceph-configure-script-job 和 credential-manager 作业：

kubectl -n uipath-infra get job "rook-ceph-configure-script-job" -o json | jq 'del(. | .spec.selector, .spec.template.metadata.labels)' | kubectl replace --force -f -
kubectl -n uipath-infra get job "credential-manager-job" -o json | jq 'del(. | .spec.selector, .spec.template.metadata.labels)' | kubectl replace --force -f -
kubectl -n uipath delete pod -l app.kubernetes.io/component=orchestratorkubectl -n uipath-infra get job "rook-ceph-configure-script-job" -o json | jq 'del(. | .spec.selector, .spec.template.metadata.labels)' | kubectl replace --force -f -
kubectl -n uipath-infra get job "credential-manager-job" -o json | jq 'del(. | .spec.selector, .spec.template.metadata.labels)' | kubectl replace --force -f -
kubectl -n uipath delete pod -l app.kubernetes.io/component=orchestrator