- 概述
- 要求
- 安装
- 安装后
- 集群管理
- 监控和警示
- 使用监控堆栈
- 警示 runbook
- 迁移和升级
- 特定于产品的配置
- 最佳实践和维护
- 故障排除
警示 runbook
|
警示严重性 |
描述 |
|---|---|
|
信息 | 意外但无害。可以静默，但在诊断期间可能有用。 |
|
Warning | 表明功能有针对性地降级或在不久的将来可能降级，这可能会影响整个集群。建议迅速采取行动（通常在几天内），以保持集群的正常运行。 |
|
重要事项 | 已知会导致集群中普遍存在的功能严重降级。需要立即采取行动（当天）修复集群。 |
kubectl describe, and logs with kubectl logs to see details on possible crashes. If the issue persists, contact UiPath® Support.
kubectl logs 检查 Pod 日志，以查看是否有任何进度指示。如果问题仍然存在，请联系 UiPath® 支持团队。
There has been an attempted update to a deployment or statefulset, but it has failed, and a rollback has not yet occurred. Contact UiPath® Support.
In high availability clusters with multiple replicas, this alert fires when the number of replicas is not optimal. This may occur when there are not enough resources in the cluster to schedule. Check resource utilization, and add capacity as necessary. Otherwise contact UiPath® Support.
An update to a statefulset has failed. Contact UiPath® Support.
另请参阅：有状态副本集。
守护程序集推出失败。请联系 UiPath® 支持团队。
另请参阅：守护程序集。
kubectl describe of the pod for more information. The most common cause of waiting containers is a failure to pull the image. For air-gapped clusters, this could mean that the local registry is not available. If the issue persists, contact UiPath® Support.
这可能表明其中一个节点存在问题，检查每个节点的运行状况，并修复任何已知问题。否则，请联系 UiPath® 支持团队。
A job takes more than 12 hours to complete. This is not expected. Contact UiPath® Support.
A job has failed; however, most jobs are retried automatically. If the issue persists, contact UiPath® Support.
The autoscaler cannot scale the targeted resource as configured. If desired is higher than actual, then there may be a lack of resources. If desired is lower than actual, pods may be stuck while shutting down. If the issue persists, contact UiPath® Support.
另请参阅：水平 Pod 自动调节
给定服务的副本数量已达到最大值。当对集群发出的请求数量非常多时，就会发生这种情况。如果预计会有暂时的高流量，您可以静默此警示。但是，此警示表示集群已满，无法处理更多流量。如果集群上有更多资源容量可用，您可以按照以下说明增加服务的最大副本数：
# Find the horizontal autoscaler that controls the replicas of the desired resource
kubectl get hpa -A
# Increase the number of max replicas of the desired resource, replacing <namespace> <resource> and <maxReplicas>
kubectl -n <namespace> patch hpa <resource> --patch '{"spec":{"maxReplicas":<maxReplicas>}}'# Find the horizontal autoscaler that controls the replicas of the desired resource
kubectl get hpa -A
# Increase the number of max replicas of the desired resource, replacing <namespace> <resource> and <maxReplicas>
kubectl -n <namespace> patch hpa <resource> --patch '{"spec":{"maxReplicas":<maxReplicas>}}'另请参阅：水平 Pod 自动调节。
这些警告表明集群不能容忍节点故障。对于单节点评估集群，这是已知的，并且系统可能会静默这些警示。对于多节点 HA 就绪生产设置，当太多节点运行状况不佳而无法支持高可用性时，将触发这些警示，并指示应将节点恢复正常状态或进行更换。
KubeCPUQuotaOvercommit, KubeMemoryQuotaOvercommit, KubeQuotaAlmostFull, KubeQuotaFullyUsed, KubeQuotaExceeded
这些警示与通过自定义添加的命名空间资源配额有关，这些配额仅存在于集群中。命名空间资源配额不会作为 Automation Suite 安装的一部分添加。
另请参阅：资源配额。
Warning：可用空间少于 15%，可能会在四天内填满。
Critical：可用空间小于 3%。
对于空间不足的任何服务，数据可能难以恢复，因此应在可用空间达到 0％ 之前调整卷的大小。请参阅以下说明：配置集群。
对于特定于 Prometheus 的警示，请参阅 Prometheus 存储 使用情况以获取更多详细信息和说明。
The Kube State Metrics collector is not able to collect metrics from the cluster without errors. This means important alerts may not fire. Contact UiPath® Support.
另请参阅：发布时的 Kube 状态指标。
Warning：用于对 Kubernetes API 服务器进行身份验证的客户端证书将在七天内过期。
Critical：用于对 Kubernetes API 服务器进行身份验证的客户端证书将在一天内过期。
您必须续订证书。
指示 Kubernetes 控制平面存在问题。检查主节点的运行状况，解决所有未解决的问题，如果问题持续存在，请联系 UiPath 支持团队。
另请参阅：
指定节点上运行的 Pod 过多。
Warning：Kubelet 的客户端或服务器证书将在七天内过期。
Critical：Kubelet 的客户端或服务器证书将在一天内过期。
您必须续订证书。
特定节点上的文件系统正在填满。通过添加磁盘或装载未使用的磁盘来配置更多空间。
节点上的物理网络接口有问题。如果问题仍然存在，则可能需要更换。
当集群接近配置的内存和存储限制时，这些警示会发出警告。 这可能发生在最近使用量大幅增加的集群上（通常来自机器人而不是用户），或者在未调整 Prometheus 资源的情况下将节点添加到集群中时。 这是因为要收集的指标数量增加。
可以在 Kubernetes/持久卷仪表板上看到存储利用率的提高率：
您可以按照此处的说明，通过调整 PVC 的大小来调整 PVC： 配置集群。
可以在 Kubernetes/计算资源/Pod 仪表板上看到内存利用率的增加率。
您可以通过在 ArgoCD 的 rancher-monitoring 应用程序中编辑 Prometheus 内存资源限制来进行调整。单击“保存”后，rancher-monitoring 应用程序将自动重新同步。
请注意，Prometheus 需要一些时间才能重新启动并再次开始在 Grafana 中显示指标。通常情况下，耗时不到 10 分钟，即使是大型集群也不例外。
这些是具有多个警示管理器副本的 HA 集群的内部警示管理器错误。警示可能会间歇性地出现和消失。暂时缩小规模，然后扩大警示管理器副本可能会解决此问题。
要解决此问题，请执行以下步骤：
-
缩放至零。请注意，Pod 需要一段时间才能关闭：
kubectl scale statefulset -n cattle-monitoring-system alertmanager-rancher-monitoring-alertmanager --replicas=0kubectl scale statefulset -n cattle-monitoring-system alertmanager-rancher-monitoring-alertmanager --replicas=0 -
缩小到 2：
kubectl scale statefulset -n cattle-monitoring-system alertmanager-rancher-monitoring-alertmanager --replicas=2kubectl scale statefulset -n cattle-monitoring-system alertmanager-rancher-monitoring-alertmanager --replicas=2 -
检查 Alertmanager Pod 是否已启动以及是否处于正在运行状态：
kubectl get po -n cattle-monitoring-systemkubectl get po -n cattle-monitoring-system
如果问题仍然存在，请联系 UiPath® 支持团队。
PrometheusOperatorListErrors, PrometheusOperatorWatchErrors, PrometheusOperatorSyncFailed, PrometheusOperatorReconcileErrors, PrometheusOperatorNodeLookupErrors, PrometheusOperatorNotReady, PrometheusOperatorRejectedResources
Internal errors of the Prometheus operator, which controls Prometheus resources. Prometheus itself may still be healthy while these errors are present; however, this error indicates there is degraded monitoring configurability. Contact UiPath® Support.
Prometheus has failed to load or reload configuration. Please check any custom Prometheus configurations for input errors. Otherwise contact UiPath® Support.
PrometheusErrorSendingAlertsToSomeAlertmanagers, PrometheusErrorSendingAlertsToAnyAlertmanager, PrometheusNotConnectedToAlertmanagers
The connection from Prometheus to AlertManager is not healthy. Metrics may still be queryable, and Grafana dashboards may still show them, but alerts will not fire. Check any custom configuration of AlertManager for input errors and and otherwise contact UiPath® Support.
PrometheusNotificationQueueRunningFull, PrometheusTSDBReloadsFailing, PrometheusTSDBCompactionsFailing, PrometheusNotIngestingSamples, PrometheusDuplicateTimestamps, PrometheusOutOfOrderTimestamps, PrometheusRemoteStorageFailures, PrometheusRemoteWriteBehind, PrometheusRemoteWriteDesiredShards
Internal Prometheus errors indicating metrics may not be collected as expected. Please contact UiPath® Support.
This may happen if there are malformed alerts based on non-existent metrics or incorrect PromQL syntax. Contact UiPath® Support if no custom alerts have been added.
Prometheus 无法评估是否应触发警示。如果警示太多，可能会发生这种情况。请删除昂贵的自定义警示评估和/或查看有关增加 Prometheus CPU 限制的文档。如果未添加自定义警示，请联系 UiPath® 支持团队。
UiPathAvailabilityHighTrafficUserFacing, UiPathAvailabilityHighTrafficBackend, UiPathAvailabilityMediumTrafficUserFacing, UiPathAvailabilityMediumTrafficBackend, UiPathAvailabilityLowTrafficUserFacing, UiPathAvailabilityLowTrafficBackend
The number of http 500 responses from UiPath® services exceeds a given threshold.
|
流量级别 |
20 分钟内的请求数 |
错误阈值（适用于 HTTP 500） |
|---|---|---|
|
高 |
>100,000 |
0.1% |
|
中 |
10,000 到 100,000 之间 |
1% |
|
低 |
< 10,000 |
5% |
面向用户的服务中的错误可能会导致可在 Automation Suite 用户界面中直接观察到的功能降级，而后端服务中的错误则不会产生明显的后果。
警示会指明哪个服务的错误率较高。要了解报告服务所依赖的其他服务可能存在哪些级联问题，您可以使用 Istio 工作负载仪表板，该仪表板会显示服务之间的错误。
Please double check any recently reconfigured Automation Suite products. Detailed logs are also available with the kubectl logs command. If the error persists, please contact UiPath® Support.
This job obtains the latest Kerberos ticket from the AD server for SQL-integrated authentication. Failures in this job would cause SQL server authentication to fail. Please contact UiPath® Support.
- 警示严重性键
- 常规.规则
- TargetDown
- Watchdog
- kubernetes-apps
- KubePodCrashLooping
- KubePodNotReady
- KubeDeploymentGenerationMismatch, KubeStatefulSetGenerationMismatch
- KubeDeploymentReplicasMismatch, KubeStatefulSetReplicasMismatch
- KubeStatefulSetUpdateNotRolledOut
- KubeDaemonSetRolloutStuck
- KubeContainerWaiting
- KubeDaemonSetNotScheduled, KubeDaemonSetMisScheduled
- KubeJobCompletion
- KubeJobFailed
- KubeHpaReplicasMismatch
- KubeHpaMaxedOut
- kubernetes-resources
- KubeCPUOvercommit, KubeMemoryOvercommit
- KubeCPUQuotaOvercommit, KubeMemoryQuotaOvercommit, KubeQuotaAlmostFull, KubeQuotaFullyUsed, KubeQuotaExceeded
- CPUThrottlingHigh
- Kubernetes-storage
- KubePersistentVolumeFillingUp
- KubePersistentVolumeErrors
- kube-state-metrics
- KubeStateMetricsListErrors, KubeStateMetricsWatchErrors
- kubernetes-system-apiserver
- KubeClientCertificateExpiration
- AggregatedAPIErrors, AggregatedAPIDown, KubeAPIDown, KubeAPITerminatedRequests
- kubernetes-system-kubelet
- KubeNodeNotReady, KubeNodeUnreachable, KubeNodeReadinessFlapping, KubeletPlegDurationHigh, KubeletPodStartUpLatencyHigh, KubeletDown
- KubeletTooManyPods
- KubeletClientCertificateExpiration, KubeletServerCertificateExpiration
- KubeletClientCertificateRenewalErrors, KubeletServerCertificateRenewalErrors
- kubernetes-system
- KubeVersionMismatch
- KubeClientErrors
- Kube-apiserver-slos
- KubeAPIErrorBudgetBurn
- node-exporter
- NodeFilesystemSpaceFillingUp、NodeFilesystemAlmostOutOfSpace、NodeFilesystemFilesFillingUp
- NodeRAIDDegraded
- NodeRAIDDiskFailure
- NodeNetworkReceiveErrs, NodeNetworkTransmitErrs, NodeHighNumberConntrackEntriesUsed
- NodeClockSkewDetected, NodeClockNotSynchronising
- node-network
- NodeNetworkInterfaceFlapping
- uipath.prometheus.resource.provisioning.alerts
- PrometheusMemoryUsage, PrometheusStorageUsage
- alertmanager.rules
- AlertmanagerConfigInconsistent
- AlertmanagerFailedReload
- prometheus-operator
- PrometheusOperatorListErrors, PrometheusOperatorWatchErrors, PrometheusOperatorSyncFailed, PrometheusOperatorReconcileErrors, PrometheusOperatorNodeLookupErrors, PrometheusOperatorNotReady, PrometheusOperatorRejectedResources
- Prometheus
- PrometheusBadConfig
- PrometheusErrorSendingAlertsToSomeAlertmanagers, PrometheusErrorSendingAlertsToAnyAlertmanager, PrometheusNotConnectedToAlertmanagers
- PrometheusNotificationQueueRunningFull, PrometheusTSDBReloadsFailing, PrometheusTSDBCompactionsFailing, PrometheusNotIngestingSamples, PrometheusDuplicateTimestamps, PrometheusOutOfOrderTimestamps, PrometheusRemoteStorageFailures, PrometheusRemoteWriteBehind, PrometheusRemoteWriteDesiredShards
- PrometheusRuleFailures
- PrometheusMissingRuleEvaluations
- PrometheusTargetLimitHit
- uipath.availability.alerts
- UiPathAvailabilityHighTrafficUserFacing, UiPathAvailabilityHighTrafficBackend, UiPathAvailabilityMediumTrafficUserFacing, UiPathAvailabilityMediumTrafficBackend, UiPathAvailabilityLowTrafficUserFacing, UiPathAvailabilityLowTrafficBackend
- uipath.cronjob.alerts.rules
- UiPath CronJob“kerberos-tgt-refresh”失败
- UiPath CronJob Kerberos-tgt-secret-update 失败
- Osd-alert.rules
- CephOSDNearFull
- CephOSDCriticallyFull
