- Overview
- Requirements
- Recommended: Deployment templates
- Manual: Preparing the installation
- Manual: Preparing the installation
- Step 1: Configuring the OCI-compliant registry for offline installations
- Step 2: Configuring the external objectstore
- Step 3: Configuring High Availability Add-on
- Step 4: Configuring Microsoft SQL Server
- Step 5: Configuring the load balancer
- Step 6: Configuring the DNS
- Step 7: Configuring the disks
- Step 8: Configuring kernel and OS level settings
- Step 9: Configuring the node ports
- Step 10: Applying miscellaneous settings
- Step 12: Validating and installing the required RPM packages
- Step 13: Generating cluster_config.json
- Certificate configuration
- Database configuration
- External Objectstore configuration
- Pre-signed URL configuration
- External OCI-compliant registry configuration
- Disaster recovery: Active/Passive and Active/Active configurations
- High Availability Add-on configuration
- Orchestrator-specific configuration
- Insights-specific configuration
- Process Mining-specific configuration
- Document Understanding-specific configuration
- Automation Suite Robots-specific configuration
- Monitoring configuration
- Optional: Configuring the proxy server
- Optional: Enabling resilience to zonal failures in a multi-node HA-ready production cluster
- Optional: Passing custom resolv.conf
- Optional: Increasing fault tolerance
- install-uipath.sh parameters
- Adding a dedicated agent node with GPU support
- Adding a dedicated agent Node for Task Mining
- Connecting Task Mining application
- Adding a Dedicated Agent Node for Automation Suite Robots
- Step 15: Configuring the temporary Docker registry for offline installations
- Step 16: Validating the prerequisites for the installation
- Manual: Performing the installation
- Post-installation
- Cluster administration
- Managing products
- Getting Started with the Cluster Administration portal
- Migrating objectstore from persistent volume to raw disks
- Migrating from in-cluster to external High Availability Add-on
- Migrating data between objectstores
- Migrating in-cluster objectstore to external objectstore
- Migrating to an external OCI-compliant registry
- Switching to the secondary cluster manually in an Active/Passive setup
- Disaster Recovery: Performing post-installation operations
- Converting an existing installation to multi-site setup
- Guidelines on upgrading an Active/Passive or Active/Active deployment
- Guidelines on backing up and restoring an Active/Passive or Active/Active deployment
- Redirecting traffic for the unsupported services to the primary cluster
- Monitoring and alerting
- Migration and upgrade
- Step 1: Moving the Identity organization data from standalone to Automation Suite
- Step 2: Restoring the standalone product database
- Step 3: Backing up the platform database in Automation Suite
- Step 4: Merging organizations in Automation Suite
- Step 5: Updating the migrated product connection strings
- Step 6: Migrating standalone Orchestrator
- Step 7: Migrating standalone Insights
- Step 8: Deleting the default tenant
- B) Single tenant migration
- Migrating from Automation Suite on Linux to Automation Suite on EKS/AKS
- Upgrading Automation Suite
- Downloading the installation packages and getting all the files on the first server node
- Retrieving the latest applied configuration from the cluster
- Updating the cluster configuration
- Configuring the OCI-compliant registry for offline installations
- Executing the upgrade
- Performing post-upgrade operations
- Product-specific configuration
- Using the Orchestrator Configurator Tool
- Configuring Orchestrator parameters
- Orchestrator appSettings
- Configuring appSettings
- Configuring the maximum request size
- Overriding cluster-level storage configuration
- Configuring credential stores
- Configuring encryption key per tenant
- Cleaning up the Orchestrator database
- Best practices and maintenance
- Troubleshooting
- How to troubleshoot services during installation
- How to uninstall the cluster
- How to clean up offline artifacts to improve disk space
- How to clear Redis data
- How to enable Istio logging
- How to manually clean up logs
- How to clean up old logs stored in the sf-logs bucket
- How to disable streaming logs for AI Center
- How to debug failed Automation Suite installations
- How to delete images from the old installer after upgrade
- How to disable TX checksum offloading
- How to upgrade from Automation Suite 2022.10.10 and 2022.4.11 to 2023.10.2
- How to manually set the ArgoCD log level to Info
- How to expand AI Center storage
- How to generate the encoded pull_secret_value for external registries
- How to address weak ciphers in TLS 1.2
- Unable to run an offline installation on RHEL 8.4 OS
- Error in downloading the bundle
- Offline installation fails because of missing binary
- Certificate issue in offline installation
- First installation fails during Longhorn setup
- SQL connection string validation error
- Prerequisite check for selinux iscsid module fails
- Azure disk not marked as SSD
- Failure after certificate update
- Antivirus causes installation issues
- Automation Suite not working after OS upgrade
- Automation Suite requires backlog_wait_time to be set to 0
- Volume unable to mount due to not being ready for workloads
- Support bundle log collection failure
- Test Automation SQL connection string is ignored
- Single-node upgrade fails at the fabric stage
- Cluster unhealthy after automated upgrade from 2021.10
- Upgrade fails due to unhealthy Ceph
- RKE2 not getting started due to space issue
- Volume unable to mount and remains in attach/detach loop state
- Upgrade fails due to classic objects in the Orchestrator database
- Ceph cluster found in a degraded state after side-by-side upgrade
- Unhealthy Insights component causes the migration to fail
- Service upgrade fails for Apps
- In-place upgrade timeouts
- Docker registry migration stuck in PVC deletion stage
- AI Center provisioning failure after upgrading to 2023.10
- Upgrade fails in offline environments
- SQL validation fails during upgrade
- snapshot-controller-crds pod in CrashLoopBackOff state after upgrade
- Longhorn REST API endpoint upgrade/reinstall error
- Setting a timeout interval for the management portals
- Authentication not working after migration
- Kinit: Cannot find KDC for realm <AD Domain> while getting initial credentials
- Kinit: Keytab contains no suitable keys for *** while getting initial credentials
- GSSAPI operation failed due to invalid status code
- Alarm received for failed Kerberos-tgt-update job
- SSPI provider: Server not found in Kerberos database
- Login failed for AD user due to disabled account
- ArgoCD login failed
- Update the underlying directory connections
- Failure to get the sandbox image
- Pods not showing in ArgoCD UI
- Redis probe failure
- RKE2 server fails to start
- Secret not found in UiPath namespace
- ArgoCD goes into progressing state after first installation
- MongoDB pods in CrashLoopBackOff or pending PVC provisioning after deletion
- Unhealthy services after cluster restore or rollback
- Pods stuck in Init:0/X
- Missing Ceph-rook metrics from monitoring dashboards
- Pods cannot communicate with FQDN in a proxy environment
- Running High Availability with Process Mining
- Process Mining ingestion failed when logged in using Kerberos
- After Disaster Recovery Dapr is not working properly for Process Mining and Task Mining
- Unable to connect to AutomationSuite_ProcessMining_Warehouse database using a pyodbc format connection string
- Airflow installation fails with sqlalchemy.exc.ArgumentError: Could not parse rfc1738 URL from string ''
- How to add an IP table rule to use SQL Server port 1433
- Running the diagnostics tool
- Using the Automation Suite Support Bundle Tool
- Exploring Logs
Multi-node architecture and design consideration
The following architecture diagram depicts a deployment of Automation Suite on Linux with Kubernetes installed on six machines, a load balancer, and data storage. There are multiple machine types: three server nodes, two agent nodes, and one specialized agent node.
etcd
component, which is part of the Kubernetes Control Plane. For more details, see the etcd documentation. For the same reason, the majority of server nodes must be available at any point to keep the cluster healthy.
These nodes also host the components that require data storage on the nodes, such as Prometheus, in-cluster objectstore Ceph, UiPath Insights, and in-cluster Docker registry.
Agent nodes are sometimes called worker nodes. The purpose of these nodes is to host UiPath® services and other shared suite capabilities. Since there is no data disk attached to these nodes, they cannot host the components that require disk storage.
Agent nodes do not impose any restriction on the number of the nodes to be available at any point in time. As long as the resulting cluster has enough capacity to host all the pods from the lost nodes, the cluster will work as expected without any disruption.
These nodes are the special agent nodes dedicated to special tasks, such as the Task Mining node for analysis, Automation Suite Robots node for robots execution, and the GPU node for the Document Understanding model. You cannot host other UiPath® services on these nodes.
The load balancer, which is installed outside Automation Suite, acts as an entry point for accessing applications hosted on the Automation Suite cluster. The load balancer is required to stand against node fault tolerance. All server nodes are required to be configured on the load balancer, but agent nodes can also be configured optionally. However, specialized agent nodes are not required.
When robots try to access Orchestrator, the call lands on the load balancer, and then it is passed to any of the available nodes. Each node also hosts the networking component called Istio, which is a service mesh that also acts like a load balancer. When the call is received by Istio running on the node, it tries to locate the Orchestrator instance on the entire cluster. Once it is found, it will redirect the call to that instance.
It entirely depends on you whether you go for more smaller machines or fewer larger machines, with both options having their own pros and cons. A higher number of smaller machines provides better resilience to node fault tolerance compared to a smaller number of larger machines. At the same time, it also introduces additional management overhead.
For example, if your Automation Suite cluster requires a 96 vCPU, then you can opt for either of the following options:
-
Option 1: 6 machines of 16vCPU each.
-
Impact: Losing a machine only reduces the capacity of the cluster by 16 vCPU, so it only impacts services if the resulting cluster does not have capacity to host all the pods. However, managing 6 machines implies a larger effort.
-
-
Option 2: 3 machines of 32vCPU each
-
Impact: Losing a machine reduces the capacity of the cluster by 32vCPU, which has a major impact on Automation Suite. However, managing 3 machines implies a smaller effort.
-
To conclude, the deployment design depends on the goal. If the goal is better fault tolerance, then more smaller machines are the choice. However, if the goal is less management overhead, then the smaller number of larger machines should be the choice.
Whether you opt for all server nodes instead of agent nodes depends on your RTO or RPO.
For instance, let's say that your Automation Suite needs 80 vCPU. You can achieve this as follows:
-
Option 1: 5 server machine with 16vCPU each. Here you can lose at most 2 server nodes.
-
Recommended if the goal is resilience to data loss. Even if 2 server nodes are lost, data will be intact and can be rebuilt from the remaining replicas.
-
-
Option 2: 3 servers nodes and the 2 agent nodes with 16vCPU each. Here you can lose 1 server node and both the agent nodes, so a total 3 machines.
-
Recommended if the goal is resilience to node availability. Even without 3 machines, the cluster will still be available with limited capability, and once the nodes are back, the entire cluster will be recovered. However this setup is more prone to data loss because of the storage attached to the sever nodes. If 2 server nodes are entirely lost, then it may be difficult to rebuild the data again without restoring it from the backup.
-