automation-suite
2023.10
false
UiPath logo, featuring letters U and I in white

Automation Suite on Linux Installation Guide

Last updated Feb 3, 2025

Cron jobs triggering hundreds of jobs causing high load on the nodes

Description

Before 2023.10.6, 2023.4.9, and 2022.10.14, some Automation Suite cron jobs did not set Concurrency Policy to Forbid. During node maintenance events, this issue can trigger hundreds of cron jobs at the same time, leading to a high load on the node. To learn how to solve this issue, see Solution 1.

The issue can also cause failures in pod scheduling due to a Cilium error: Key allocation attempt failed attempt=0 error="no more available IDs in configured space". To learn how to solve this issue, see Solution 2.

Solution 1

Identify which cron jobs are triggering the high number of instances and delete those cron jobs.

CRONJOB_NAME="<cronjob-name>"
NAMESPACE="<namespace>"

# Delete all jobs associated with the CronJob
kubectl get jobs -n "$NAMESPACE" | grep "$CRONJOB_NAME" | awk '{print $1}' | xargs -I{} kubectl delete job -n "$NAMESPACE" {}CRONJOB_NAME="<cronjob-name>"
NAMESPACE="<namespace>"

# Delete all jobs associated with the CronJob
kubectl get jobs -n "$NAMESPACE" | grep "$CRONJOB_NAME" | awk '{print $1}' | xargs -I{} kubectl delete job -n "$NAMESPACE" {} 

Solution 2

You can verify the failures in pod scheduling by running this command: kubectl get ciliumid | wc -l.

If the command returns more than 10000 IDs, you need to clean up unused Cilium IDs and restart Cilium deployments.

# Step 1: Get all CiliumIDs
ALL_CILIUMIDS=$(kubectl get ciliumid --no-headers -o custom-columns=":metadata.name")

# Step 2: Get active pod UIDs 
ACTIVE_POD_UIDS=$(kubectl get pods -A -o jsonpath='{.items[*].metadata.uid}') 

# Step 3: Compare and delete stale CiliumIDs 
for CID in $ALL_CILIUMIDS; do
 if [[ ! "$ACTIVE_POD_UIDS" =~ "$CID" ]]; then
  echo "Deleting stale CiliumID: $CID"
  kubectl delete ciliumid "$CID"
 fi
done

# Step 4: Restart cilium operator and daemonset
kubectl rollout restart deployment/cilium-operator -n kube-system
kubectl rollout restart daemonset/cilium -n kube-system# Step 1: Get all CiliumIDs
ALL_CILIUMIDS=$(kubectl get ciliumid --no-headers -o custom-columns=":metadata.name")

# Step 2: Get active pod UIDs 
ACTIVE_POD_UIDS=$(kubectl get pods -A -o jsonpath='{.items[*].metadata.uid}') 

# Step 3: Compare and delete stale CiliumIDs 
for CID in $ALL_CILIUMIDS; do
 if [[ ! "$ACTIVE_POD_UIDS" =~ "$CID" ]]; then
  echo "Deleting stale CiliumID: $CID"
  kubectl delete ciliumid "$CID"
 fi
done

# Step 4: Restart cilium operator and daemonset
kubectl rollout restart deployment/cilium-operator -n kube-system
kubectl rollout restart daemonset/cilium -n kube-system
  • Description
  • Solution 1
  • Solution 2

Was this page helpful?

Get The Help You Need
Learning RPA - Automation Courses
UiPath Community Forum
Uipath Logo White
Trust and Security
© 2005-2025 UiPath. All rights reserved.