- Overview
- Requirements
- Installation
- Post-installation
- Cluster administration
- Monitoring and alerting
- Migration and upgrade
- Product-specific configuration
- Best practices and maintenance
- Troubleshooting
- How to Troubleshoot Services During Installation
- How to Uninstall the Cluster
- How to clean up offline artifacts to improve disk space
- How to disable TLS 1.0 and 1.1
- How to enable Istio logging
- How to manually clean up logs
- How to clean up old logs stored in the sf-logs bundle
- How to debug failed Automation Suite installations
- How to disable TX checksum offloading
- Unable to run an offline installation on RHEL 8.4 OS
- Error in Downloading the Bundle
- Offline installation fails because of missing binary
- Certificate issue in offline installation
- SQL connection string validation error
- Failure After Certificate Update
- Automation Suite Requires Backlog_wait_time to Be Set 1
- Cannot Log in After Migration
- Setting a timeout interval for the management portals
- Update the underlying directory connections
- Kinit: Cannot Find KDC for Realm <AD Domain> While Getting Initial Credentials
- Kinit: Keytab Contains No Suitable Keys for *** While Getting Initial Credentials
- GSSAPI Operation Failed With Error: An Invalid Status Code Was Supplied (Client's Credentials Have Been Revoked).
- Login Failed for User <ADDOMAIN><aduser>. Reason: The Account Is Disabled.
- Alarm Received for Failed Kerberos-tgt-update Job
- SSPI Provider: Server Not Found in Kerberos Database
- Failure to get the sandbox image
- Pods not showing in ArgoCD UI
- Redis Probe Failure
- RKE2 Server Fails to Start
- Secret Not Found in UiPath Namespace
- ArgoCD goes into progressing state after first installation
- Unexpected Inconsistency; Run Fsck Manually
- Missing Self-heal-operator and Sf-k8-utils Repo
- Degraded MongoDB or Business Applications After Cluster Restore
- Unhealthy Services After Cluster Restore or Rollback
- Using the Automation Suite Diagnostics Tool
- Using the Automation Suite Support Bundle Tool
- Exploring Logs
Using the Automation Suite Diagnostics Tool
The Automation Suite Diagnostics Tool is the first thing to use when facing any issues with Automation Suite. It checks the health of different required components and gives a consolidated report.
supportability-tools
zip and extract its contents using the following commands:
curl "https://download.uipath.com/automation-suite/2021.10.3/supportability-tools-2021.10.3.zip" -o supportability-tools-2021.10.3.zip
unzip supportability-tools-2021.10.3.zip -d support-tools
support-tools/diagnostics-tool/
folder using the bash diagnostics-report.sh
command.
The following table lists out the checks the Automation Suite Diagnostics Tool performs. Note that you can run the script on any of the nodes in the cluster as well as externally.
Node |
Checks |
---|---|
Master node |
|
Agent node |
|
External machine |
Note: To run the script from an external machine, first set the proper
kubeconfig context to the cluster, and then pass the -e flag to the script bash diagnostics-report.sh -e .
|
Sample report generated by the Automation Suite Diagnostics Tool.
INFO logs in green show that the required checks passed. However, you should still properly check the disk/memory usage to avoid hidden errors.
Even though these messages do not signal a high risk, you might have to rectify them, as they might be affecting some services in certain scenarios.
You must fix the issues described by these messages as they impact some service in the cluster.
If these services are down, it means the node is down. Try restarting the service using systemctl restart <service-name> as this should fix the issue.
/var/lib
as Kubernetes uses it to store its data. If the directory is full, various issues might arise. To prevent these problems,
make sure to increase its size.
For all the nodes, we specify if they are under Disk Pressure or Memory Pressure. If that happens, workloads on these nodes might start showing issues. Check if there are any other processes running on these nodes that are consuming resources and remove them if that is the case.
We use Ceph as S3 Object storage for storing logs and files from different applications. You can view the status of its services. If they are down, you might have to restart them. Make sure to also check if the disk usage by Ceph is full.
443
and 31443
to be open with the hostname that was provided. The report indicates if they are not accessible. Make sure to open the appropriate
ports if pointed here.
The tool checks if the uploaded certificate is valid for the given hostname and if it has not expired. If the certificate does not meet these criteria, errors occur. To prevent this, make sure to check your uploaded certificate and change it if required.
Since some services require GPU to be present on some of the nodes in the cluster, the Automation Suite Diagnostics Tool checks if there is are GPU nodes and prints number of such nodes. If you are expecting GPU nodes to be present and they do not show up here, that means something went wrong in GPU setup.
MongoDB is an important component that the UiPath Apps service uses. If either MongoDB or its primary instance is down, you need to investigate the issue using the support bundle.
RabbitMQ and DockerRegistry are two important components that some services use. If any of them is down, you need to investigate the issue and a restart.
ArgoCD is our application lifecycle management (ALM) tool. If any of its services are down, then other applications may become outdated or have other issues. Recovering these services is important, and might need further debugging.
The Automation Suite Diagnostics Tool shows whether ArgoCD applications are missing and degraded.
- If applications are missing, go to the ArgoCD UI and sync it.
- If applications are degraded, additional debugging is needed to investigate the errors thrown by ArgoCD
- Reading Diagnostics Reports
- INFO Logs
- WARN Messages
- ERROR Messages
- Rke2-server or Rke2-agent Service Down
- Directory Size Mounted at /var/lib
- Rke2 Version
- Disk Pressure or Memory Pressure
- Ceph Services Status
- Ports 443 and 31443
- Certificate Validity
- GPU
- MongoDB
- RabbitMQ and DockerRegistry
- ArgoCD Services Down
- Missing or Degraded ArgoCD Applications