- Overview
- Requirements
- Installation
- Post-installation
- Cluster administration
- Managing products
- Managing the cluster in ArgoCD
- Setting up the external NFS server
- Automated: Enabling the Backup on the Cluster
- Automated: Disabling the Backup on the Cluster
- Automated, Online: Restoring the Cluster
- Automated, Offline: Restoring the Cluster
- Manual: Enabling the Backup on the Cluster
- Manual: Disabling the Backup on the Cluster
- Manual, Online: Restoring the Cluster
- Manual, Offline: Restoring the Cluster
- Additional configuration
- Migrating objectstore from persistent volume to raw disks
- Monitoring and alerting
- Migration and upgrade
- Migration options
- Step 1: Moving the Identity organization data from standalone to Automation Suite
- Step 2: Restoring the standalone product database
- Step 3: Backing up the platform database in Automation Suite
- Step 4: Merging organizations in Automation Suite
- Step 5: Updating the migrated product connection strings
- Step 6: Migrating standalone Insights
- Step 7: Deleting the default tenant
- B) Single tenant migration
- Product-specific configuration
- Best practices and maintenance
- Troubleshooting
- How to Troubleshoot Services During Installation
- How to Uninstall the Cluster
- How to clean up offline artifacts to improve disk space
- How to clear Redis data
- How to enable Istio logging
- How to manually clean up logs
- How to clean up old logs stored in the sf-logs bundle
- How to disable streaming logs for AI Center
- How to debug failed Automation Suite installations
- How to delete images from the old installer after upgrade
- How to automatically clean up Longhorn snapshots
- How to disable NIC checksum offloading
- Unable to run an offline installation on RHEL 8.4 OS
- Error in Downloading the Bundle
- Offline installation fails because of missing binary
- Certificate issue in offline installation
- First installation fails during Longhorn setup
- SQL connection string validation error
- Prerequisite check for selinux iscsid module fails
- Azure disk not marked as SSD
- Failure After Certificate Update
- Automation Suite not working after OS upgrade
- Automation Suite Requires Backlog_wait_time to Be Set 1
- Volume unable to mount due to not being ready for workloads
- RKE2 fails during installation and upgrade
- Failure to upload or download data in objectstore
- PVC resize does not heal Ceph
- Failure to Resize Objectstore PVC
- Rook Ceph or Looker pod stuck in Init state
- StatefulSet volume attachment error
- Failure to create persistent volumes
- Storage reclamation patch
- Backup failed due to TooManySnapshots error
- All Longhorn replicas are faulted
- Setting a timeout interval for the management portals
- Update the underlying directory connections
- Cannot Log in After Migration
- Kinit: Cannot Find KDC for Realm <AD Domain> While Getting Initial Credentials
- Kinit: Keytab Contains No Suitable Keys for *** While Getting Initial Credentials
- GSSAPI Operation Failed With Error: An Invalid Status Code Was Supplied (Client's Credentials Have Been Revoked).
- Alarm Received for Failed Kerberos-tgt-update Job
- SSPI Provider: Server Not Found in Kerberos Database
- Login Failed for User <ADDOMAIN><aduser>. Reason: The Account Is Disabled.
- ArgoCD login failed
- Failure to get the sandbox image
- Pods not showing in ArgoCD UI
- Redis Probe Failure
- RKE2 Server Fails to Start
- Secret Not Found in UiPath Namespace
- After the Initial Install, ArgoCD App Went Into Progressing State
- MongoDB pods in CrashLoopBackOff or pending PVC provisioning after deletion
- Unexpected Inconsistency; Run Fsck Manually
- Degraded MongoDB or Business Applications After Cluster Restore
- Missing Self-heal-operator and Sf-k8-utils Repo
- Unhealthy Services After Cluster Restore or Rollback
- RabbitMQ pod stuck in CrashLoopBackOff
- Prometheus in CrashloopBackoff state with out-of-memory (OOM) error
- Missing Ceph-rook metrics from monitoring dashboards
- Using the Automation Suite Diagnostics Tool
- Using the Automation Suite Support Bundle Tool
- Exploring Logs
Storage reclamation patch
To ensure high performance and prevent outages, we optimize the cluster storage for Automation Suite by reclaiming unused storage. However, ocasionally some of the actively used storage may be reclaimed, leading to service impact for multi-node clusters and possible data loss for single-node clusters.
The following versions are affected by this issue:
- 2021.10.3 to 2021.10.10
- 2022.4.8 and earlier
- 2022.10.7 and earlier
- 2023.4.2 and earlier
To make sure actively used storage on nodes is not reclaimed by Automation Suite, run the following script:
#!/bin/bash
echo ""
echo "Starting Storage Reclamation Patch"
echo ""
echo "Checking that this is a server node"
if [ $(sudo systemctl is-enabled rke2-server) ]; then
echo " This is a server node"
echo ""
else
echo " FATAL: This is not a server node"
echo " This script should only be run on a server node"
echo "Exiting script"
echo ""
exit 1
fi
echo "Generating patch.yaml file at: /tmp/patch.yaml"
if [ -f /tmp/patch.yaml ]; then
echo " FATAL: Patch file: /tmp/patch.yaml file already exists"
echo " Remove existing /tmp/patch.yaml file and re-run script"
echo " Command to remove file: sudo rm -rf /tmp/patch.yaml"
echo "Exiting script"
echo ""
exit 1
fi
sudo cat <<'EOF' > /tmp/patch.yaml
spec:
template:
spec:
containers:
- name: longhorn-replica-folder-cleanup
args:
- /host
- /bin/bash
- -ec
- |
while true;
do
set -o pipefail
export KUBECONFIG=/etc/rancher/rke2/rke2.yaml PATH=$PATH:/var/lib/rancher/rke2/bin:$INSTALLER_PATH
which kubectl >> /dev/null || {
echo "kubectl not found"
exit 1
}
which jq >> /dev/null || {
echo "jq not found"
exit 1
}
directories=$(find ${LONGHORN_DISK_PATH}/replicas/ -maxdepth 1 -mindepth 1 -type d)
for dir in $directories;
do
basename=$(basename "$dir")
volume_name=${basename%-*}
replica_name=$(kubectl -n longhorn-system get replicas.longhorn.io -o json | jq --arg dir "$basename" '.items[] | select(.spec.dataDirectoryName==$dir) | .metadata.name')
if kubectl -n longhorn-system get volumes.longhorn.io "$volume_name" &>/dev/null;
then
if [[ -z ${replica_name} ]];
then
robust_status=$(kubectl -n longhorn-system get volumes.longhorn.io "$volume_name" -o jsonpath='{.status.robustness}')
if [[ "${robust_status}" == "healthy" || "${robust_status}" == "degraded" ]];
then
echo "Replica not found but Volume found with a valid status (robust status ${robust_status}). Data directory $dir can be deleted"
rm -rf $dir
else
echo "Replica not found but Volume found with robust status ${robust_status}. Need to check if there is still a valid replica before deleting data directory $dir so that the directory is not required for recovery"
fi
else
echo "Volume found and there is a replica using the data directory $dir"
fi
else
if kubectl -n longhorn-system get volumes.longhorn.io "$volume_name" 2>&1 | grep "NotFound";
then
echo "Volume object not found. Data directory $dir can be deleted."
rm -rf $dir
else
echo "Could not fetch volume for $dir"
fi
fi
done
sleep 600
done
EOF
# Checker to see if patch.yaml file was created
if [ -f /tmp/patch.yaml ]; then
echo " /tmp/patch.yaml file created"
echo ""
else
echo " FATAL: /tmp/patch.yaml file not created"
echo " Previous command did not run successfully. Try running: sudo touch /tmp/patch.yaml to see why the command failed to generate the file /tmp/patch.yaml"
echo " If help is needed, please contact UiPath Support"
echo "Exiting script"
echo ""
exit 1
fi
echo "Applying patch.yaml file"
echo ' Executing the command: sudo /var/lib/rancher/rke2/bin/kubectl --kubeconfig /etc/rancher/rke2/rke2.yaml -n kube-system patch daemonset longhorn-replica-folder-cleanup --patch "$(cat /tmp/patch.yaml)"'
sudo /var/lib/rancher/rke2/bin/kubectl --kubeconfig /etc/rancher/rke2/rke2.yaml -n kube-system patch daemonset longhorn-replica-folder-cleanup --patch "$(cat /tmp/patch.yaml)" 1>/dev/null
exit_code=$?
echo ""
echo "Checking that the patch was applied"
if [ $exit_code -eq 0 ]; then
echo " Patch was applied successfully"
echo ""
else
echo " FATAL: Patch was not applied successfully"
echo " Previous command did not run successfully. Check previous errors and Contact UiPath Support for help."
echo " Before re-running script please run this command: sudo rm -f /tmp/patch.yaml"
echo "Exiting script"
echo ""
exit 1
fi
echo "Removing /tmp/patch.yaml file"
echo ""
sudo rm -rf /tmp/patch.yaml
echo "System is patched. Make sure to run this tool in all environments on any master node"
echo ""
#!/bin/bash
echo ""
echo "Starting Storage Reclamation Patch"
echo ""
echo "Checking that this is a server node"
if [ $(sudo systemctl is-enabled rke2-server) ]; then
echo " This is a server node"
echo ""
else
echo " FATAL: This is not a server node"
echo " This script should only be run on a server node"
echo "Exiting script"
echo ""
exit 1
fi
echo "Generating patch.yaml file at: /tmp/patch.yaml"
if [ -f /tmp/patch.yaml ]; then
echo " FATAL: Patch file: /tmp/patch.yaml file already exists"
echo " Remove existing /tmp/patch.yaml file and re-run script"
echo " Command to remove file: sudo rm -rf /tmp/patch.yaml"
echo "Exiting script"
echo ""
exit 1
fi
sudo cat <<'EOF' > /tmp/patch.yaml
spec:
template:
spec:
containers:
- name: longhorn-replica-folder-cleanup
args:
- /host
- /bin/bash
- -ec
- |
while true;
do
set -o pipefail
export KUBECONFIG=/etc/rancher/rke2/rke2.yaml PATH=$PATH:/var/lib/rancher/rke2/bin:$INSTALLER_PATH
which kubectl >> /dev/null || {
echo "kubectl not found"
exit 1
}
which jq >> /dev/null || {
echo "jq not found"
exit 1
}
directories=$(find ${LONGHORN_DISK_PATH}/replicas/ -maxdepth 1 -mindepth 1 -type d)
for dir in $directories;
do
basename=$(basename "$dir")
volume_name=${basename%-*}
replica_name=$(kubectl -n longhorn-system get replicas.longhorn.io -o json | jq --arg dir "$basename" '.items[] | select(.spec.dataDirectoryName==$dir) | .metadata.name')
if kubectl -n longhorn-system get volumes.longhorn.io "$volume_name" &>/dev/null;
then
if [[ -z ${replica_name} ]];
then
robust_status=$(kubectl -n longhorn-system get volumes.longhorn.io "$volume_name" -o jsonpath='{.status.robustness}')
if [[ "${robust_status}" == "healthy" || "${robust_status}" == "degraded" ]];
then
echo "Replica not found but Volume found with a valid status (robust status ${robust_status}). Data directory $dir can be deleted"
rm -rf $dir
else
echo "Replica not found but Volume found with robust status ${robust_status}. Need to check if there is still a valid replica before deleting data directory $dir so that the directory is not required for recovery"
fi
else
echo "Volume found and there is a replica using the data directory $dir"
fi
else
if kubectl -n longhorn-system get volumes.longhorn.io "$volume_name" 2>&1 | grep "NotFound";
then
echo "Volume object not found. Data directory $dir can be deleted."
rm -rf $dir
else
echo "Could not fetch volume for $dir"
fi
fi
done
sleep 600
done
EOF
# Checker to see if patch.yaml file was created
if [ -f /tmp/patch.yaml ]; then
echo " /tmp/patch.yaml file created"
echo ""
else
echo " FATAL: /tmp/patch.yaml file not created"
echo " Previous command did not run successfully. Try running: sudo touch /tmp/patch.yaml to see why the command failed to generate the file /tmp/patch.yaml"
echo " If help is needed, please contact UiPath Support"
echo "Exiting script"
echo ""
exit 1
fi
echo "Applying patch.yaml file"
echo ' Executing the command: sudo /var/lib/rancher/rke2/bin/kubectl --kubeconfig /etc/rancher/rke2/rke2.yaml -n kube-system patch daemonset longhorn-replica-folder-cleanup --patch "$(cat /tmp/patch.yaml)"'
sudo /var/lib/rancher/rke2/bin/kubectl --kubeconfig /etc/rancher/rke2/rke2.yaml -n kube-system patch daemonset longhorn-replica-folder-cleanup --patch "$(cat /tmp/patch.yaml)" 1>/dev/null
exit_code=$?
echo ""
echo "Checking that the patch was applied"
if [ $exit_code -eq 0 ]; then
echo " Patch was applied successfully"
echo ""
else
echo " FATAL: Patch was not applied successfully"
echo " Previous command did not run successfully. Check previous errors and Contact UiPath Support for help."
echo " Before re-running script please run this command: sudo rm -f /tmp/patch.yaml"
echo "Exiting script"
echo ""
exit 1
fi
echo "Removing /tmp/patch.yaml file"
echo ""
sudo rm -rf /tmp/patch.yaml
echo "System is patched. Make sure to run this tool in all environments on any master node"
echo ""
Running the script:
To run the script, take the following steps:
- Copy the script to one of the automation suite server nodes and name it
storageReclamationPatch.sh
. - Change the permissions to make the script executable:
chmod 755 storageReclamationPatch.sh
- Execute the script:
- Make sure that scripting is enabled so that the execution and output of the script can be captured in a log file. If any issue in case any issue is encountered, this will help our support team diagnose the issue.
- To start the scripting program and the execute the patch script run the following command:
script storageReclamationPatch.log ./storageReclamationPatch.sh
- Use the
exit
command to exit the script and to generate thestorageReclamationPatch.log
log file. If you encounter any issues during this stage, share them with our support team.
Example of running the script:
[admin_1@autosuite storageReclamationPatch]$ script storageReclamationPatch.log
Script started, file is storageReclamationPatch.log
[admin_1@autosuite storageReclamationPatch]$ ./storageReclamationPatch.sh
//Script executes, maybe some debugging is done
[admin_1@autosuite storageReclamationPatch]$ exit
exit
Script done, file is storageReclamationPatch.log
[admin_1@autosuite storageReclamationPatch]$ script storageReclamationPatch.log
Script started, file is storageReclamationPatch.log
[admin_1@autosuite storageReclamationPatch]$ ./storageReclamationPatch.sh
//Script executes, maybe some debugging is done
[admin_1@autosuite storageReclamationPatch]$ exit
exit
Script done, file is storageReclamationPatch.log
If executed successfully, the script displays the following message:
Checking that the patch was applied Patch was applied successfully
storageReclamationPatch.log
file with support.