运行诊断工具

运行 Automation Suite 诊断工具，以生成集群运行状况报告，并识别 Automation Suite 中的常见问题。

Automation Suite 诊断工具会运行一系列检查，以生成有关集群运行状况的报告，您可以分析该报告，以识别问题及其潜在的根本原因。该工具可帮助您查找常见问题，例如数据库连接丢失或凭据无效或过期。

Automation Suite 诊断工具在 uipathctl 和 uipathtools 中均可用，您可以在管理计算机上下载该工具。

uipathtools 是一个 CLI 工具，其中包含特定于运行状况命令的 uipathctl 功能的子集。该工具向后兼容，适用于任何受支持的 Automation Suite 版本。如果您遇到任何问题，我们建议首先使用 uipathtools 。

快速验证

快速验证

check 和 test 命令可让您快速了解集群的状态，而无需运行深度分析。

• check 依赖于 ArgoCD 运行状况和同步状态，不会修改集群中的任何状态
• test 会调查应用程序、部署或 Pod，并临时改变集群的状态，为您提供这些见解。

运行状况检查

要运行运行状况检查，请根据您使用的 CLI 工具使用以下命令之一：

• 如果使用 uipathctl，请运行：

  ./bin/uipathctl health check
  

• 如果使用 uipathtools，请运行：

  ./bin/uipathtools health check
  

生成的报告的示例输出：

INFO[0038] Found 3 pods for etcd
INFO[0038] Running the health command - [etcdctl endpoint health --endpoints https://localhost:2379 --cacert /var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --cert /var/lib/rancher/rke2/server/tls/etcd/server-client.crt --key /var/lib/rancher/rke2/server/tls/etcd/server-client.key]
INFO[0039] No credentials provided for registry: registry.uipath.com
INFO[0039] Checking if pods for component argocd-server exist
INFO[0039] Checking if pods for component argocd-repo-server exist
INFO[0039] Found 2 pods for Istio
INFO[0039] Checking if pods for component argocd-application-controller exist
INFO[0039] Checking if pods for component redis-ha exist
INFO[0040] application actioncenter-actions has sync enabled
INFO[0040] application actioncenter-bupproxyservice has sync enabled
INFO[0040] application actioncenter-processes has sync enabled
INFO[0040] application ai-app-deployment has sync enabled
INFO[0040] application ai-appmanager-deployment has sync enabled
INFO[0040] application ai-deployer-deployment has sync enabled
INFO[0040] application ai-helper-deployment has sync enabled
INFO[0040] application ai-pkgmanager-deployment has sync enabled
INFO[0040] application ai-trainer-deployment has sync enabled
INFO[0040] application aievents-deploy has sync enabled
INFO[0040] application ailoadbalancer-cleanup has sync enabled
INFO[0040] application ailoadbalancer-service has sync enabled
INFO[0040] application aimetering has sync enabled
INFO[0040] application airflow-scheduler has sync enabled
INFO[0040] application airflow-statsd has sync enabled
INFO[0040] application airflow-webserver has sync enabled
INFO[0040] application aistorage has sync enabled
INFO[0040] application aistorage-cleanup has sync enabled
INFO[0040] application apps-designer has sync enabled
INFO[0040] application apps-runtime has sync enabled
INFO[0040] application apps-server has sync enabled
INFO[0040] application apps-signalr has sync enabled
INFO[0040] application asrobots has sync enabled
INFO[0040] application auth-dex has sync enabled
INFO[0040] application auth-oauth2-proxy has sync enabled
INFO[0040] application automationhub-ah-frontdoor-service has sync enabled
INFO[0040] application automationhub-ah-open-api-service has sync enabled
INFO[0040] application automationhub-ah-tenant-service has sync enabled
INFO[0040] application automationhub-ah-web-client has sync enabled
INFO[0040] application automationsolutions has sync enabled
INFO[0040] application datapipeline-api has sync enabled
INFO[0040] application dataservice-designer has sync enabled
INFO[0040] application dataservice-runtime has sync enabled
INFO[0040] application dataservice-taskrunner has sync enabled
INFO[0040] application du-aimodelhost-2404 has sync enabled
INFO[0040] application du-aimodelhost-classifier-2404 has sync enabled
INFO[0040] application du-annotations has sync enabled
INFO[0040] application du-annotations-background-tasks has sync enabled
INFO[0040] application du-app-service has sync enabled
INFO[0040] application du-audit-cleanup has sync enabled
INFO[0040] application du-audit-service has sync enabled
INFO[0040] application du-classifier has sync enabled
INFO[0040] application du-deployments has sync enabled
INFO[0040] application du-digitizer has sync enabled
INFO[0040] application du-digitizer-cleanup has sync enabled
INFO[0040] application du-digitizer-worker-deployment has sync enabled
INFO[0040] application du-document-processor-cleanup has sync enabled
INFO[0040] application du-document-processor-service has sync enabled
INFO[0040] application du-document-types-service has sync enabled
INFO[0040] application du-document-types-service-cleanup has sync enabled
INFO[0040] application du-documentmanager-dm has sync enabled
INFO[0040] application du-documents-service has sync enabled
INFO[0040] application du-documents-service-cleanup has sync enabled
INFO[0040] application du-extended-ocr has sync enabled
INFO[0040] application du-extended-ocr-proxy has sync enabled
INFO[0040] application du-framework has sync enabled
INFO[0040] application du-framework-cleanup has sync enabled
INFO[0040] application du-framework-worker has sync enabled
INFO[0040] application du-frontend has sync enabled
INFO[0040] application du-measure-service has sync enabled
INFO[0040] application du-ocr has sync enabled
INFO[0040] application du-provisioning has sync enabled
INFO[0040] application du-services-fe has sync enabled
INFO[0040] application du-services-ikc has sync enabled
INFO[0040] application du-ssde has sync enabled
INFO[0040] application du-training-classifier-2404 has sync enabled
INFO[0040] application du-training-service has sync enabled
INFO[0040] application du-training-service-cleanup has sync enabled
INFO[0040] application du-training-ssde-2404 has sync enabled
INFO[0040] application identity-service-api has sync enabled
INFO[0040] application identity-service-web has sync enabled
INFO[0040] application insights-insightsportal has sync enabled
INFO[0040] application insights-insightsprovisioning has sync enabled
INFO[0040] application notificationcoreworker has sync enabled
INFO[0040] application notificationserviceapi has sync enabled
INFO[0040] application orchestrator has sync enabled
INFO[0040] application platform-authorization-service has sync enabled
INFO[0040] application platform-license-accountant has sync enabled
INFO[0040] application platform-license-accountant-worker has sync enabled
INFO[0040] application platform-license-resource-manager has sync enabled
INFO[0040] application platform-license-resource-manager-worker has sync enabled
INFO[0040] application platform-location-service has sync enabled
INFO[0040] application platform-messagebus-service has sync enabled
INFO[0040] application platform-organization-management-service has sync enabled
INFO[0040] application platform-portal has sync enabled
INFO[0040] application platform-resource-catalog-service has sync enabled
INFO[0040] application process-mining has sync enabled
INFO[0040] application process-mining-dbt-exec has sync enabled
INFO[0040] application process-mining-frontend has sync enabled
INFO[0040] application process-mining-technology-webapi has sync enabled
INFO[0040] application process-mining-technology-workerservice has sync enabled
INFO[0040] application publishermetaservice has sync enabled
INFO[0040] application pushgateway-prometheus-pushgateway has sync enabled
INFO[0040] application reloader-reloader has sync enabled
INFO[0040] application robotube has sync enabled
INFO[0040] application studio-governance-api has sync enabled
INFO[0040] application studio-governance-web has sync enabled
INFO[0040] application testmanager has sync enabled
INFO[0040] application usergroupresolverworker has sync enabled
INFO[0040] application usersubscriptionservice has sync enabled
INFO[0040] application webhook-service has sync enabled
INFO[0040] Pod etcd-server0 is healthy
INFO[0040] Running the health command - [etcdctl endpoint health --endpoints https://localhost:2379 --cacert /var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --cert /var/lib/rancher/rke2/server/tls/etcd/server-client.crt --key /var/lib/rancher/rke2/server/tls/etcd/server-client.key]
INFO[0041] Pod etcd-server1 is healthy
INFO[0041] Running the health command - [etcdctl endpoint health --endpoints https://localhost:2379 --cacert /var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --cert /var/lib/rancher/rke2/server/tls/etcd/server-client.crt --key /var/lib/rancher/rke2/server/tls/etcd/server-client.key]
INFO[0042] Pod etcd-server2 is healthy
INFO[0044] Waited for job uipath-check/grafana-vj8m5-curl-podfzvvb to reach status COMPLETED, desiredStatus [COMPLETED] with message CompletionsReached
INFO[0044] status COMPLETED is contained in desiredStatus [COMPLETED]
INFO[0044] Querying for running pod in namespace uipath-check, and job.Name grafana-vj8m5-curl-podfzvvb
INFO[0060] Fetching from helm-credential URL in argocd namespace
INFO[0061] Fetching from helm-credential URL in argocd namespace
INFO[0064] There were errors when reading cluster config: cannot parse proxy enabled: strconv.ParseBool: parsing "": invalid syntax
Ran cluster/ checks...
    Ran ACTIONCENTER checks...
        ✔ successful
    Ran AICENTER checks...
        ✔ successful
    Ran AIEVENTS checks...
        ✔ successful
    Ran AIMETERING checks...
        ✔ successful
    Ran AIRFLOW checks...
        ✔ successful
    Ran ARGOCD checks...
        ✔ successful
    Ran ASROBOTS checks...
        ✔ successful
    Ran AUTOMATIONHUB checks...
        ✔ successful
    Ran AUTOMATIONOPS checks...
        ✔ successful
    Ran AUTOMATIONSOLUTIONS checks...
        ✔ successful
    Ran BA checks...
        ✔ successful
    Ran CERT-MANAGER checks...
        ✔ successful
    Ran CILIUM checks...
        ✔ successful
    Ran DATAPIPELINE-API checks...
        ✔ successful
    Ran DATASERVICE checks...
        ✔ successful
    Ran DOCUMENTUNDERSTANDING checks...
        ✔ successful
    Ran ETCD checks...
        ✔ successful
    Ran GATEKEEPER checks...
        ✔ successful
    Ran GRAFANA checks...
        ✔ successful
    Ran INSIGHTS checks...
        ✔ successful
    Ran ISTIO checks...
        ✔ successful
    Ran LOGGING checks...
        ✔ successful
    Ran MAINTENANCE checks...
        ✔ successful
    Ran NODE checks...
        ✔ successful
    Ran NOTIFICATIONSERVICE checks...
        ✔ successful
    Ran ORCHESTRATOR checks...
        ✔ successful
    Ran PLATFORM checks...
        ✔ successful
    Ran POD checks...
        ✔ successful
    Ran PROCESSMINING checks...
        ✔ successful
    Ran RELOADER checks...
        ✔ successful
    Ran REPLICAS checks...
        ✔ successful
    Ran ROBOTUBE checks...
        ✔ successful
    Ran SFCORE checks...
        ✔ successful
    Ran TESTMANAGER checks...
        ✔ successful
    Ran WEBHOOK checks...
        ✔ successful
Checks complete!

默认情况下， health check 命令会检查所有组件的运行状况。但是，它也允许您严格检查您感兴趣的组件：

• 如果要从执行中排除组件，请使用 --excluded 标志。

  例如，如果您不想检查 SQL 的运行状况，请运行以下命令：

  ./bin/uipathctl health check --excluded SQL
  

  该命令会检查所有组件的运行状况，SQL 除外。

• 如果您只想在执行中包含某些组件，请使用 --included 标志。

  例如，如果您只想检查 DNS 和对象存储的运行状况，请运行以下命令：

  ./bin/uipathctl health check --included DNS,OBJECTSTORAGE
  

分析日志

1. 运行检查运行状况检查后，日志显示 Data Service 应用程序的运行状况检查失败。

   ❌ [DATASERVICE]
       ❌ [DATASERVICE_HEALTH] Application health check failed: health status is Progressing and sync status is Synced
   

2. 经过进一步调查，很明显，Data Service 应用程序失败是因为 dataservice-runtime-8f5bb7d56-v5krg 和 dataservice-taskrunner-787df76c74-98h5l Pod 处于失败状态。 如果进一步分析，您会发现缺少的 dataservice-external-storage-secret 缺失。

   ❌ [POD]
       ✔ [LIST_NAMESPACES] Retrieved 25 namespaces to check pod health
       ❌ [CANNOT_MOUNT_VOLUME] Pod uipath/dataservice-runtime-8f5bb7d56-v5krg cannot mount volume: MountVolume.SetUp failed for volume "external-storage-creds" : secret "dataservice-external-storage-secret" not found
       ❌ [CANNOT_MOUNT_VOLUME] Pod uipath/dataservice-runtime-8f5bb7d56-xs9t5 cannot mount volume: MountVolume.SetUp failed for volume "external-storage-creds" : secret "dataservice-external-storage-secret" not found
       ❌ [CANNOT_MOUNT_VOLUME] Pod uipath/dataservice-taskrunner-787df76c74-98h5l cannot mount volume: MountVolume.SetUp failed for volume "external-storage-creds" : secret "dataservice-external-storage-secret" not found
   

3. 要解决此问题，请确保已在 cluster_config.json 中为对象存储提供正确的凭据。

运行状况测试

要运行运行状况测试，请使用以下命令之一，具体取决于您使用的 CLI 工具：

• 如果使用 uipathctl，请运行：

  ./bin/uipathctl health test
  

• 如果使用 uipathtools，请运行：

  ./bin/uipathtools health test
  

默认情况下，health test 命令会对所有组件执行运行状况测试。但是，它也允许您严格检查您感兴趣的组件：

• 如果要从执行中排除组件，请使用 --excluded 标志。

  例如，如果您不想检查 SQL 的运行状况，请运行以下命令：

  ./bin/uipathctl health test --excluded SQL
  

  该命令会检查所有组件的运行状况，SQL 除外。

• 如果您只想在执行中包含某些组件，请使用 --included 标志。

  例如，如果您只想检查 DNS 和对象存储的运行状况，请运行以下命令：

  ./bin/uipathctl test --included DNS,OBJECTSTORAGE
  

备注：

如果比较 Data Service 应用程序的 check 和 test 命令的输出，您可以看到前者验证应用程序的运行状况，而后者则检查路由。

已知问题

您可能会收到类似于以下示例的错误消息。 您可以忽略它，因为您无需执行任何 Actions 。

E0621 23:32:56.426321   24470 reflector.go:138] external/io_k8s_client_go/tools/cache/reflector.go:167: Failed to watch *v1.Pod: context deadline exceeded
E0621 23:32:56.426392   24470 reflector.go:138] external/io_k8s_client_go/tools/cache/reflector.go:167: Failed to watch *v1.Pod: context deadline exceeded
E0621 23:32:56.444420   24470 reflector.go:138] external/io_k8s_client_go/tools/cache/reflector.go:167: Failed to watch *v1.Pod: context deadline exceeded
E0621 23:32:56.446150   24470 reflector.go:138] external/io_k8s_client_go/tools/cache/reflector.go:167: Failed to watch *v1.Pod: context deadline exceeded
E0621 23:32:56.513357   24470 reflector.go:138] external/io_k8s_client_go/tools/cache/reflector.go:167: Failed to watch *v1.Pod: context deadline exceeded

深度验证

深度验证

diagnose命令可用于深入了解集群状态。 它可以帮助您识别各个级别的问题，例如 SQL、对象存储、节点、密码、Istio、网络等。

• 它涵盖了 check 和 test 命令。
• 它会运行在安装 Automation Suite 之前执行的先决条件检查，以验证在安装后对环境配置进行的更改以及可能导致问题的更改。
• 它在所有节点上运行，以收集任何特定于节点的问题，例如资源不可用、任何网络干扰等。

要运行诊断检查，请使用以下命令之一，具体取决于您使用的 CLI 工具：

• 如果使用 uipathctl，请运行：

  ./bin/uipathctl health diagnose cluster_config.json --versions versions/helm-charts.json
  

• 如果使用 uipathtools，请运行：

  ./bin/uipathtools health diagnose cluster_config.json --versions versions/helm-charts.json
  

备注：

上述示例已缩减。 实际日志包含更多信息。 您可能会注意到， diagnose 命令在多个级别运行，例如基础架构、网络、存储、Pod、DNS 等。

分析日志

您可以在以前的日志中注意到两个潜在的问题：

• Istio 的配置错误，可能会导致访问 Document Understanding 平台时出现问题：

  ❌ [ISTIO]
      ✔ [ISTIO_SYNC_STATUS] Istio sync is up-to-date
      ❌ [ISTIO_ENVOY_CONFIG_STATUS] Istio Envoy configs are not healthy: Error [IST0101] (VirtualService uipath/du-platform-vs) Referenced host:port not found: "aistorage:5000"
  

• Data Service 不可用。 请参阅代码示例中的 Ceph。

  ❌ [DATASERVICE]
      ❌ [DATASERVICE_HEALTH] Application health check failed: health status is Progressing and sync status is Synced
  ❌ [CANNOT_MOUNT_VOLUME] Pod uipath/dataservice-runtime-8f5bb7d56-v5krg cannot mount volume: (combined from similar events): Unable to attach or mount volumes: unmounted volumes=[external-storage-creds], unattached volumes=[workload-socket is-secrets openssl istio-podinfo temp-location cert-location istio-data external-storage-creds workload-certs istio-envoy java domain-cert-config edk2 credential-socket tmp additional-ca-cert-config pem istiod-ca-cert istio-token app-secrets ceph-storage-creds]: timed out waiting for the condition
      ❌ [CANNOT_MOUNT_VOLUME] Pod uipath/dataservice-runtime-8f5bb7d56-xs9t5 cannot mount volume: MountVolume.SetUp failed for volume "external-storage-creds" : secret "dataservice-external-storage-secret" not found
      ❌ [CANNOT_MOUNT_VOLUME] Pod uipath/dataservice-taskrunner-787df76c74-98h5l cannot mount volume: MountVolume.SetUp failed for volume "external-storage-creds" : secret "dataservice-external-storage-secret" not found
  

已知问题

您可能会收到类似于以下示例的错误消息。 您可以忽略它，因为您无需执行任何 Actions 。

其他实用程序

其他实用程序

所有 Automation Suite 诊断工具命令（check、 test和 diagnose）都支持其他筛选和输出格式。

下表描述了诊断工具的可用筛选器，并提供了用法示例。

筛选条件	描述	用法
--included	要包含在验证中的服务的逗号分隔列表	./bin/uipathctl health diagnose cluster_config.json --versions versions/helm-charts.json --included ISTIO,INSIGHTS 此命令仅针对 Istio 和 Insights 运行诊断。
--excluded	要从验证中排除的服务的逗号分隔列表	./bin/uipathctl health test --excluded ISTIO,INSIGHTS 此命令在整个集群中运行测试，Istio 和 Insights 除外。

输出格式

Automation Suite 诊断工具可以生成多种格式的报告： json、 yaml、 text和 junit。 您可以通过 --output 标志将这些值传递给任何命令。 当您要利用这些工具在其基础上构建自己的故障排除框架时，这些输出格式非常方便。

用法示例

用法	示例输出
./bin/uipathctl health check --included DATASERVICE --output json ./bin/uipathtools health check --included DATASERVICE --output json	{ "cluster/": { "DATASERVICE": [ { "name": "DATASERVICE_HEALTH", "description": "Application health check failed: health status is Progressing and sync status is Synced", "status": "failed" } ] } }
./bin/uipathctl health check --included DATASERVICE --output yaml ./bin/uipathtools health check --included DATASERVICE --output yaml	--- locationType: cluster DATASERVICE: - name: DATASERVICE_HEALTH description: "Application health check failed: health status is Progressing and sync status is Synced" status: failed
./bin/uipathctl health check --included DATASERVICE --output text ./bin/uipathtools health check --included DATASERVICE --output text	Checks run on cluster/ ❌ [DATASERVICE] ❌ [DATASERVICE_HEALTH] Application health check failed: health status is Progressing and sync status is Synced
./bin/uipathctl health check --included DATASERVICE --output junit ./bin/uipathtools health check --included DATASERVICE --output junit	<testsuite name="Health" tests="1" errors="0" failures="1" time="0" timestamp="2023-06-22T01:59:08.313362+05:30" hostname=""> <testcase name="DATASERVICE_HEALTH" classname="" time="0"> <failure message="Application health check failed: health status is Progressing and sync status is Synced" type=""></failure> </testcase> </testsuite>

读取诊断报告

信息日志

绿色的 INFO 日志显示所需的检查已通过。但是，您仍应正确检查磁盘/内存的使用情况，以避免隐藏的错误。

警告消息

即使这些消息并不表示高风险，您也可能需要纠正它们，因为在某些情况下它们可能会影响某些服务。

错误消息

您必须修复这些消息描述的问题，因为它们会影响集群中的某些服务。

Rke2-server 或 Rke2-agent 服务关闭

如果这些服务关闭，则意味着节点已关闭。尝试使用 systemctl restart <service-name> 命令来重新启动服务，因为这应该可以解决问题。

装载于 /var/lib 的目录大小

该报告显示在 /var/lib 中装载的目录大小，因为 Kubernetes 使用它来存储其数据。如果目录已满，则可能会出现各种问题。为防止出现这些问题，请确保增加其大小。

Rke2 版本

报告将显示 rke2 版本以供参考。

磁盘压力或内存压力

对于所有节点，我们指定它们是处于磁盘压力之下，还是内存压力之下。如果发生这种情况，这些节点上的工作负载可能会开始出现问题。检查这些节点上是否正在运行任何其他正在消耗资源的流程，如果是这种情况，请将其删除。

Ceph 服务状态

我们使用 Ceph 作为 S3 对象存储，用于存储来自不同应用程序的日志和文件。您可以查看其服务的状态。如果它们已关闭，则可能必须重新启动它们。请务必同时检查 Ceph 的磁盘使用情况是否已满。

端口 443 和 31443

我们希望使用提供的主机名打开 443 和 31443 端口。报告会指出它们是否不可访问。如果指向此处，请确保打开相应的端口。

证书有效性

该工具会检查上传的证书对于给定的主机名是否有效，以及是否未过期。如果证书不符合这些条件，则会发生错误。为防止出现这种情况，请务必检查您上传的证书，并在需要时进行更改。

GPU

由于某些服务要求集群中的某些节点存在 GPU，因此 Automation Suite 诊断工具会检查是否存在 GPU 节点并打印此类节点的数量。如果您期望 GPU 节点存在，但它们没有显示在此处，则意味着 GPU 设置中出现了问题。

Docker 注册表

DockerRegistry 是某些服务使用的重要组件。如果关闭，则需要调查问题并重新启动。

ArgoCD 服务关闭

ArgoCD 是我们的应用程序生命周期管理 (ALM) 工具。如果其任何服务关闭，则其他应用程序可能已过期或存在其他问题。恢复这些服务很重要，并且可能需要进一步调试。

ArgoCD 应用程序缺失或降级

Automation Suite 诊断工具显示 ArgoCD 应用程序是否丢失和降级。

• 如果缺少应用程序，请转到 ArgoCD 用户界面并进行同步。
• 如果应用程序降级，则需要额外调试以调查 ArgoCD 引发的错误