Storage alerts

kubernetes-system

KubernetesDiskPressure

This alert indicates that disk usage is very high on the Kubernetes node.

If this alert fires, try to see which pod is consuming more disk:

• Confirm if the node is under DiskPressure using the following command:

  kubectl describe node <node-name>

  Identify for the DiskPressure condition in the output.

• Check the disk space usage on the affected node:

  df -h

  This shows disk usage on all mounted file systems. Identify where the high usage.

• If the disk is full and cleanup is insufficient, consider resizing the disk for the node (especially in cloud environments such as AWS or GCP). This process may involve expanding volumes, depending on your infrastructure.

KubernetesMemoryPressure

This alert indicates that memory usage is very high on the Kubernetes node.

The Kubernetes nodes with MemoryPressure incident type occurs when a Kubernetes cluster node is running low on memory, which can be caused by a memory leak in an application. This incident type requires immediate attention to prevent any downtime and ensure the proper functioning of the Kubernetes cluster.

If this alert fires, try to identify the pod on the node that is consuming more memory, by taking these steps:

1. Retrieve the nodes CPU and memory stats:

   kubectl top node

2. Retrieve the pods running on the node:

   kubectl get pods --all-namespaces -o wide --field-selector spec.nodeName=${NODE_NAME}

3. Check the memory usage for pods in a namespace using:

   kubectl top pod --namespace <namespace>
   kubectl logs -f <pod-name> -n <ns>

If you are able to identify any pod with high memory usage, check the logs of the pod and look for memory leak errors.

To address the issue, increase the memory spec for the nodes if possible.

If the issue persists, generate the support bundle and contact UiPath® Support.

KubePersistentVolumeFillingUp

When Warning: The available space is less than 30% and is likely to fill up within four days.

When Critical: The available space is less than 10%.

For any services that run out of space, data may be difficult to recover, so volumes should be resized before hitting 0% available space.

For instructions, see Configuring the cluster.

For Prometheus-specific alerts, see PrometheusStorageUsage for more details and instructions.

KubePersistentVolumeErrors

PersistentVolume is not able to be provisioned. This means any service requiring the volume would not start. Check for other errors with Longhorn and/or Ceph storage and contact UiPath® Support.

node-exporter

NodeFilesystemSpaceFillingUp

The filesystem on a particular node is filling up.

If this alert fires, consider the following steps:

• Confirm if the node is under DiskPressure using the following command:

  kubectl describe node <node-name>

  Identify for the DiskPressure condition in the output.

• Clear the logs and temporary files. Check for large log files in /var/log/ and clean them, if possible.

• Check the disk space usage on the affected node:

  df -h

  This shows disk usage on all mounted file systems. Identify where the high usage.

• If the disk is full and cleanup is insufficient, consider resizing the disk for the node (especially in cloud environments such as AWS or GCP). This process may involve expanding volumes, depending on your infrastructure.

NodeFilesystemAlmostOutOfSpace

The filesystem on a particular node is filling up. Provision more space by adding a disk or mounting unused disks.

NodeFilesystemFilesFillingUp

The filesystem on a particular node is filling up. Provision more space by adding a disk or mounting unused disks.

NodeFilesystemAlmostOutOfFiles

The filesystem on a particular node is filling up. Provision more space by adding a disk or mounting unused disks.

NodeNetworkReceiveErrs

These errors indicate that the network driver is reporting a high number of failures. This can be caused by physicall hardware failures, or misconfiguration in the physical network. This issue pertains to the OS and is not controlled by the UiPath® application.

The alert is triggered by monitoring the/proc/net/dev counter that the linux kernel provides.

Contact your network admin and the team that manages the physical infrastructure.

NodeNetworkTransmitErrs

These errors indicate that the network driver is reporting a high number of failures. This can be caused by physicall hardware failures, or misconfiguration in the physical network. This issue pertains to the OS and is not controlled by the UiPath® application.

The alert is triggered by monitoring the/proc/net/dev counter that the linux kernel provides.

Contact your network admin and the team that manages the physical infrastructure.

ceph.rules, cluster-state-alert.rules

CephClusterErrorState

This alert indicates that the Ceph storage cluster has been in error state for more than 10m.

This alert reflects that the rook-ceph-mgr job has been in error state for an unacceptable amount of time. Check for other alerts that might have triggered prior to this one and troubleshoot those first.

kubectl describe cephcluster -n rook-ceph

CephMonQuorumAtRisk

This alert indicates that storage cluster quorum is low.

Multiple mons work together to provide redundancy; this is possible because each keeps a copy of the metadata. The cluster is deployed with 3 mons, and requires 2 or more mons to be up and running for quorum and for the storage operations to run. If quorum is lost, access to data is at risk.

If this alert fires, check if any OSDs are in terminating state, if there are any, force delete those pods, and wait for some time for the operator to reconcile. If the issue persists, contact UiPath® support.

CephMgrIsAbsent

This alert indicates that Ceph Manager has disappeared from Prometheus target discovery.

If this alert fires, check and ensure the the Ceph manager pod is up and running and healthy. If the pod is healthy please check the logs and check if the pod is enable to emit Prometheus metrics.

CephNodeDown

This alert indicates thata node running Ceph pods is down. While storage operations continue to function as Ceph is designed to deal with a node failure, it is recommended to resolve the issue to minimize the risk of another node going down and affecting storage functions.

If this alert fires, in case of a multi-node cluster, the pod must to be scheduled on another node. Ensure that the new osd pods in rook-ceph namespace are running and in healthy state in the new node.

You can check the node failure by describing the node using the following command:

kubectl get nodes

Check the node to identify the root cause of the issue and contact UiPath® support.

cluster-utilization-alert.rules

CephClusterNearFull

This alert indicates that the Ceph storage cluster utilization has crossed 75% and will become read-only at 85%.

If this alert fires, free up some space in Ceph by deleting some unused datasets in AI Center or Task Mining or expand the storage available for Ceph PVC.

Before resizing PVC, make sure you meet the storage requirements. For details, see Evaluating your storage needs.

CephClusterCriticallyFull

This alert indicates that Ceph storage cluster utilization has crossed 80% and will become read-only at 85%.

If this alert fires, free up some space in Ceph by deleting some unused datasets in AI Center or Task Mining or expand the storage available for Ceph PVC.

Before resizing PVC, make sure you meet the storage requirements. For details, see Evaluating your storage needs.

CephClusterReadOnly

This alert indicates that Ceph storage cluster utilization has crossed 85% and will become read-only now. Free up some space or expand the storage cluster immediately.

If this alert fires, free up some space in Ceph by deleting some unused datasets in AI Center or Task Mining or expand the storage available for Ceph PVC.

Before resizing PVC, make sure you meet the storage requirements. For details, see Evaluating your storage needs.

osd-alert.rules

CephOSDCriticallyFull

When the alert severity is Critical, the available space is less than 20%.

For any services that run out of space, data may be difficult to recover, so you should resize volumes before hitting 10% available space. See the following instructions: Configuring the cluster.

CephOSDNearFull

This alert indicates that the Ceph storage cluster utilization has crossed 75% and will become read-only at 85%.

If this alert fires, free up some space in Ceph by deleting some unused datasets in AI Center or Task Mining or expand the storage available for Ceph PVC.

Before resizing PVC, make sure you meet the storage requirements. For details, see Evaluating your storage needs.

PersistentVolumeUsageNearFull

This alert indicates that the Ceph storage cluster utilization has crossed 75% and will become read-only at 85%.

If this alert fires, free up some space in Ceph by deleting some unused datasets in AI Center or Task Mining or expand the storage available for Ceph PVC.

Before resizing PVC, make sure you meet the storage requirements. For details, see Evaluating your storage needs.

CephOSDFlapping

This alert indicates that the storage daemon has restarted more than 5 times in last 5 minutes.

If this alert fires, take the following steps:

1. Check the Ceph cluster health. Yuo must run ceph status in the Ceph toolbox to identify the flapping OSDs:

   kubectl -n rook-ceph exec -it <ceph-tools-pod> -- ceph status

   You can identify the Ceph tools pod by listing the pods in the namespace:

   kubectl -n rook-ceph get pod | grep tools

2. Check the OSD logs for the flapping OSD pod to identify issues:

   kubectl -n rook-ceph logs <osd-pod>

3. Identify node level issues:

   • Check the resource usage:

     kubectl top node <node-name>

   • Check the disk health. You need to SSH into the node and rundf -h and dmesg to check disk errors.

4. Restart the OSD pod. If the issue is transient, you need to restart the flapping OSD pod:

   kubectl -n rook-ceph delete pod <osd-pod>

5. Ensure that there are no network connectivity issues between OSDs and Ceph monitors.

6. If needed, temporarily mark the flapping OSD as out:

   ceph osd out <osd-id>

7. Continue to monitor the cluster to ensure the problem does not recur.

CephOSDDiskNotResponding

This alert indicates that the host disk device is not responding.

If this alert fires, take the following steps:

1. Check the status of the Ceph cluster. You need to confirm the overall health of the Ceph cluster and get more details about the OSD status:

   • Run the following command inside the Ceph toolbox pod:

     kubectl -n rook-ceph exec -it <ceph-tools-pod> -- ceph status

   • Identify the Ceph tools pod by listing the pods in the namespace:

     kubectl -n rook-ceph get pod | grep tools

2. Check the OSD pod status. You need to check whether the OSD pods are running. Run the following command to check all OSD pod statuses:

   kubectl -n rook-ceph get pods | grep osd

   If any OSD pod is in a CrashLoopBackOff or Pending state, that could indicate an issue with the OSD disk or the underlying node.
3. Restart the affected OSD pod. If an OSD pod is in a bad state (CrashLoopBackOff, Error, etc.), you must restart the pod to see if the issue resolves itself. Kubernetes automatically attempts to reschedule the pod.

   kubectl -n rook-ceph delete pod <osd-pod>

   The OSD pod will be restarted, and if it is a transient issue, this may resolve it.

4. Check the OSD logs. If the restart did not resolve the issue, check the OSD pod logs for more details on why the disk is not responding:

   kubectl -n rook-ceph logs <osd-pod>

   Look for disk-related errors or other issues (e.g., I/O errors, failed mounts).

5. Identify node level issues. If the OSD disk is not mounted properly or has been disconnected, you can log in to the affected node and check the disk mount status:

   ssh <node> df -h

   Look for missing or unmounted disks that Ceph is expecting. If necessary, remount the disk or replace it if it has failed.

CephOSDDiskUnavailable

This alert indicates that the Ceph OSD disk not accessible on host.

If this alert fires, take the following steps:

1. Check the status of the Ceph cluster. You need to confirm the overall health of the Ceph cluster and get more details about the OSD status:

   • Run the following command inside the Ceph toolbox pod:

     kubectl -n rook-ceph exec -it <ceph-tools-pod> -- ceph status

   • Identify the Ceph tools pod by listing the pods in the namespace:

     kubectl -n rook-ceph get pod | grep tools

2. Check the OSD pod status. You need to check whether the OSD pods are running. Run the following command to check all OSD pod statuses:

   kubectl -n rook-ceph get pods | grep osd

   If any OSD pod is in a CrashLoopBackOff or Pending state, that could indicate an issue with the OSD disk or the underlying node.
3. Restart the affected OSD pod. If an OSD pod is in a bad state (CrashLoopBackOff, Error, etc.), you must restart the pod to see if the issue resolves itself. Kubernetes automatically attempts to reschedule the pod.

   kubectl -n rook-ceph delete pod <osd-pod>

   The OSD pod will be restarted, and if it is a transient issue, this may resolve it.

4. Check the OSD logs. If the restart did not resolve the issue, check the OSD pod logs for more details on why the disk is not responding:

   kubectl -n rook-ceph logs <osd-pod>

   Look for disk-related errors or other issues (e.g., I/O errors, failed mounts).

persistent-volume-alert.rules

PersistentVolumeUsageCritical

If this alert fires, free up some space in Ceph by deleting some unused datasets in AI Center or Task Mining or expand the storage available for Ceph PVC.

Before resizing PVC, make sure you meet the storage requirements. For details, see Evaluating your storage needs.

pool-quota.rules

CephPoolQuotaBytesCriticallyExhausted

This alert indicates that Ceph storage pool usage has crossed 90%.

If this alert fires, free up some space in CEPH by deleting some unused datasets in AI Center or Task Mining or expand the storage available for Ceph PVC.

Before resizing PVC, make sure you meet the storage requirements. For details, see Evaluating your storage needs.

host-disk

LowDiskForRancherPartition

This alert indicates that the free space for the /var/lib/rancher partition is less than:

• 25% – the severity of the alert is critical

You must log into the host server and verify the disk usage. You can use commands such as df -h /var/lib/rancher to check the available disk space. If you are running low on space, consider the following options:

• Clear out the unnecessary files. Over time, log files, temporary files, orphaned data, and backups can consume a significant amount of space. Regularly cleaning up these files can help maintain disk space.

• Resize the partition. If your filesystem supports it, and if there is available unused space on your disk, you could resize the partition to give it more disk space.

• Add more disk space. If the previous ptions are not sufficient, and if your infrastructure allows it, increase the size of the disk for proper functioning of Rancher.

• Check the storage usage for any abnormally large files:

  find /var/lib/rancher -type f -exec du -h {} + | sort -rh | head -n 10

• Check for any containers that are writing huge files into the disk.

LowDiskForKubeletPartition

This alert indicates that the free space for the /var/lib/kubelet partition is less than:

• 25% – the severity of the alert is critical

If this alert fires, increase the size of the disk.

LowDiskForVarPartition

This alert indicates that the free space for the /var partition is less than:

• 25% – the severity of the alert is critical

Note:

This can happen due to accumulation of container system logs.

If this alert fires, take the following steps:

1. Check the storage usage:

   find /var/ -type f -exec du -h {} + | sort -rh | head -n 10

2. Increase the size of the disk.

LowDiskForVarLogPartition

This alert indicates that the free space for the /var/lib/var partition is less than:

• 25% – the severity of the alert is critical

If this alert fires, increase the size of the disk.