- Overview
- Requirements
- Installation
- Post-installation
- Cluster administration
- Managing products
- Managing the cluster in ArgoCD
- Setting up the external NFS server
- Automated: Enabling the Backup on the Cluster
- Automated: Disabling the Backup on the Cluster
- Automated, Online: Restoring the Cluster
- Automated, Offline: Restoring the Cluster
- Manual: Enabling the Backup on the Cluster
- Manual: Disabling the Backup on the Cluster
- Manual, Online: Restoring the Cluster
- Manual, Offline: Restoring the Cluster
- Additional configuration
- Migrating objectstore from persistent volume to raw disks
- Monitoring and alerting
- Migration and upgrade
- Migration options
- Step 1: Moving the Identity organization data from standalone to Automation Suite
- Step 2: Restoring the standalone product database
- Step 3: Backing up the platform database in Automation Suite
- Step 4: Merging organizations in Automation Suite
- Step 5: Updating the migrated product connection strings
- Step 6: Migrating standalone Insights
- Step 7: Deleting the default tenant
- Performing a single tenant migration
- Product-specific configuration
- Best practices and maintenance
- Troubleshooting
- How to Troubleshoot Services During Installation
- How to Uninstall the Cluster
- How to clean up offline artifacts to improve disk space
- How to clear Redis data
- How to enable Istio logging
- How to manually clean up logs
- How to clean up old logs stored in the sf-logs bucket
- How to disable streaming logs for AI Center
- How to debug failed Automation Suite installations
- How to delete images from the old installer after upgrade
- How to automatically clean up Longhorn snapshots
- How to disable TX checksum offloading
- How to address weak ciphers in TLS 1.2
- Unable to run an offline installation on RHEL 8.4 OS
- Error in Downloading the Bundle
- Offline installation fails because of missing binary
- Certificate issue in offline installation
- First installation fails during Longhorn setup
- SQL connection string validation error
- Prerequisite check for selinux iscsid module fails
- Azure disk not marked as SSD
- Failure After Certificate Update
- Automation Suite not working after OS upgrade
- Automation Suite Requires Backlog_wait_time to Be Set 1
- Volume unable to mount due to not being ready for workloads
- RKE2 fails during installation and upgrade
- Failure to upload or download data in objectstore
- PVC resize does not heal Ceph
- Failure to Resize Objectstore PVC
- Rook Ceph or Looker pod stuck in Init state
- StatefulSet volume attachment error
- Failure to create persistent volumes
- Storage reclamation patch
- Backup failed due to TooManySnapshots error
- All Longhorn replicas are faulted
- Setting a timeout interval for the management portals
- Update the underlying directory connections
- Cannot Log in After Migration
- Kinit: Cannot Find KDC for Realm <AD Domain> While Getting Initial Credentials
- Kinit: Keytab Contains No Suitable Keys for *** While Getting Initial Credentials
- GSSAPI Operation Failed With Error: An Invalid Status Code Was Supplied (Client's Credentials Have Been Revoked).
- Alarm Received for Failed Kerberos-tgt-update Job
- SSPI Provider: Server Not Found in Kerberos Database
- Login Failed for User <ADDOMAIN><aduser>. Reason: The Account Is Disabled.
- ArgoCD login failed
- Failure to get the sandbox image
- Pods not showing in ArgoCD UI
- Redis Probe Failure
- RKE2 Server Fails to Start
- Secret Not Found in UiPath Namespace
- After the Initial Install, ArgoCD App Went Into Progressing State
- MongoDB pods in CrashLoopBackOff or pending PVC provisioning after deletion
- Unexpected Inconsistency; Run Fsck Manually
- Degraded MongoDB or Business Applications After Cluster Restore
- Missing Self-heal-operator and Sf-k8-utils Repo
- Unhealthy Services After Cluster Restore or Rollback
- RabbitMQ pod stuck in CrashLoopBackOff
- Prometheus in CrashloopBackoff state with out-of-memory (OOM) error
- Missing Ceph-rook metrics from monitoring dashboards
- Pods cannot communicate with FQDN in a proxy environment
- Using the Automation Suite Diagnostics Tool
- Using the Automation Suite support bundle
- Exploring Logs
Before You Begin
-
Make sure you meet the Prerequisites.
-
Make sure the backup is disabled on the backup cluster. Inconsistencies will occurs if a new backup is created while cluster restore is in progress. See Disabling the backup on the cluster.
-
Make sure package wget, unzip, jq are available on all the restore nodes.
-
All external data sources must be the same (SQL Server).
-
Restart the NFS server before restoring the cluster by running the following command on the NFS server node:
systemctl restart nfs-server. -
The restore cluster should have the same FQDN as the backup cluster.
Installation type Configuration instructions Requirements Offline single-node evaluation mode
Download the following files:
sf-installer.zip– Mandatory. See sf-installer.zip for download instructions.sf-infra.tar.gz– Mandatory. See sf-infra.tar.gz for download instructions.
Offline multi-node HA-ready production mode
Download below files
sf-installer.zip– Mandatory. See sf-installer.zip for download instructions.sf-infra.tar.gz– Mandatory. See sf-infra.tar.gz for download instructions.
Preparing the Restore Operation
If you have multiple nodes in the cluster, follow these steps on all the nodes.
-
Log into any machine.
-
To create the
/opt/UiPathAutomationSuitefolder, run:# Become root sudo su - # Replace {version} with actual installer version mkdir -p /opt/UiPathAutomationSuite/{version}/installer chmod -R 777 /opt/UiPathAutomationSuite# Become root sudo su - # Replace {version} with actual installer version mkdir -p /opt/UiPathAutomationSuite/{version}/installer chmod -R 777 /opt/UiPathAutomationSuite -
To copy
restore.jsonand other installer files to the/opt/UiPathAutomationSuite/{version}folder, run the following command. For details, see Preparing the restore configuration.# Replace {version} with actual installer version cp /path/to/restore.json /opt/UiPathAutomationSuite/${version}/installer cp /path/to/sf-installer.zip /opt/UiPathAutomationSuite/${version}/installer cp /path/to/sf-infra.tar.gz /opt/UiPathAutomationSuite/${version}# Replace {version} with actual installer version cp /path/to/restore.json /opt/UiPathAutomationSuite/${version}/installer cp /path/to/sf-installer.zip /opt/UiPathAutomationSuite/${version}/installer cp /path/to/sf-infra.tar.gz /opt/UiPathAutomationSuite/${version} -
To unzip
sf-installer.zip, run:# Navigate to installer folder. Replace {version} with actual installer version cd /opt/UiPathAutomationSuite/${version}/installer unzip sf-installer.zip -d .# Navigate to installer folder. Replace {version} with actual installer version cd /opt/UiPathAutomationSuite/${version}/installer unzip sf-installer.zip -d .
Restoring Etcd on the First Server Node
To restore etcd on the first node of the cluster, run:
./install-uipath.sh -i restore.json -o output.json -r --offline-bundle "../sf-infra.tar.gz" --offline-tmp-folder /opt/UiPathAutomationSuite/tmp --install-offline-prereqs --accept-license-agreement --install-type offline
./install-uipath.sh -i restore.json -o output.json -r --offline-bundle "../sf-infra.tar.gz" --offline-tmp-folder /opt/UiPathAutomationSuite/tmp --install-offline-prereqs --accept-license-agreement --install-type offline
This command restores the node configuration only on the first server node.
Restoring Etcd on the Remaining Server Nodes
To restore etcd on the other server nodes of the cluster, run:
./install-uipath.sh -i ../restore.json -o output.json -r -j server --offline-bundle "../sf-infra.tar.gz" --offline-tmp-folder /opt/UiPathAutomationSuite/tmp --install-offline-prereqs --accept-license-agreement --install-type offline
./install-uipath.sh -i ../restore.json -o output.json -r -j server --offline-bundle "../sf-infra.tar.gz" --offline-tmp-folder /opt/UiPathAutomationSuite/tmp --install-offline-prereqs --accept-license-agreement --install-type offline
This command restores the node configuration on the remaining server nodes.
Restoring Etcd on Agent Nodes
To restore etcd on agent nodes of the cluster, run:
./install-uipath.sh -i restore.json -o output.json -r -j agent --offline-bundle "../sf-infra.tar.gz" --offline-tmp-folder /opt/UiPathAutomationSuite/tmp --install-offline-prereqs --accept-license-agreement --install-type offline
./install-uipath.sh -i restore.json -o output.json -r -j agent --offline-bundle "../sf-infra.tar.gz" --offline-tmp-folder /opt/UiPathAutomationSuite/tmp --install-offline-prereqs --accept-license-agreement --install-type offline
This command restores the node configuration on the remaining agent nodes.
Restoring Etcd on Task Mining Nodes
To restore etcd on Task Mining agent nodes of the cluster, run:
./install-uipath.sh -i restore.json -o output.json -r -j task-mining --offline-bundle "../sf-infra.tar.gz" --offline-tmp-folder /opt/UiPathAutomationSuite/tmp --install-offline-prereqs --accept-license-agreement --install-type offline
./install-uipath.sh -i restore.json -o output.json -r -j task-mining --offline-bundle "../sf-infra.tar.gz" --offline-tmp-folder /opt/UiPathAutomationSuite/tmp --install-offline-prereqs --accept-license-agreement --install-type offline
This command restores the node configuration on the Task Mining nodes.
Restoring Etcd on GPU Nodes
Before restoring the GPU machine, make sure you have followed Installing a GPU driver on the machine.
To restore etcd on GPU agent nodes of the cluster, run:
./install-uipath.sh -i restore.json -o output.json -r -j gpu --offline-bundle "../sf-infra.tar.gz" --offline-tmp-folder /opt/UiPathAutomationSuite/tmp --install-offline-prereqs --accept-license-agreement --install-type offline
./install-uipath.sh -i restore.json -o output.json -r -j gpu --offline-bundle "../sf-infra.tar.gz" --offline-tmp-folder /opt/UiPathAutomationSuite/tmp --install-offline-prereqs --accept-license-agreement --install-type offline
This command restores the node configuration on the remaining GPU nodes.
Disabling the Maintenance Mode
This step is required only if the restore is part of the rollback operation during the upgrade.
Before restoring the volume, you need to disable the maintenance mode:
./configureUiPathAS.sh disable-maintenance-mode
./configureUiPathAS.sh disable-maintenance-mode
Running Volume Restore on Any Server Node
Once the etcd restore is complete, restore the volume on any of the server node by running:
\./install-uipath.sh -i restore.json -o output.json -r --volume-restore --accept-license-agreement --install-type offline
\./install-uipath.sh -i restore.json -o output.json -r --volume-restore --accept-license-agreement --install-type offline
This command restores the volume data on the cluster.
Enabling AI Center on the Restored Cluster
After restoring an Automation Suite cluster with AI Center™ enabled, follow the steps from the Enabling AI Center on the Restored Cluster procedure.
Restoring Cluster_config.json
Following the Automation Suite cluster recovery, you may want to recover the cluster_config.json for future. You may need it when adding new nodes to the cluster, when upgrading, etc.
To restore cluster_config.json, run the following command from any of the server nodes:
./configureUiPathAS.sh config get -o ./cluster_config.json
./configureUiPathAS.sh config get -o ./cluster_config.json
The backup is not enabled on the restored cluster. If you want to enable the backup, refer to Enabling the backup on the cluster.
Adding CA certificates to the trust store
After restoring the cluster, make sure to add your CA certificates to the trust store of the restored VMs. For details, see:
- Before You Begin
- Preparing the Restore Operation
- Restoring Etcd on the First Server Node
- Restoring Etcd on the Remaining Server Nodes
- Restoring Etcd on Agent Nodes
- Restoring Etcd on Task Mining Nodes
- Restoring Etcd on GPU Nodes
- Disabling the Maintenance Mode
- Running Volume Restore on Any Server Node
- Enabling AI Center on the Restored Cluster
- Restoring Cluster_config.json
- Adding CA certificates to the trust store