- Overview
- Requirements
- Recommended: Deployment templates
- Manual: Preparing the installation
- Manual: Preparing the installation
- Step 1: Configuring the OCI-compliant registry for offline installations
- Step 2: Configuring the external objectstore
- Step 3: Configuring High Availability Add-on
- Step 4: Configuring Microsoft SQL Server
- Step 5: Configuring the load balancer
- Step 6: Configuring the DNS
- Step 7: Configuring the disks
- Step 8: Configuring kernel and OS level settings
- Step 9: Configuring the node ports
- Step 10: Applying miscellaneous settings
- Step 12: Validating and installing the required RPM packages
- Step 13: Generating cluster_config.json
- Certificate configuration
- Database configuration
- External Objectstore configuration
- Pre-signed URL configuration
- External OCI-compliant registry configuration
- Disaster recovery: Active/Passive and Active/Active configurations
- High Availability Add-on configuration
- Orchestrator-specific configuration
- Insights-specific configuration
- Process Mining-specific configuration
- Document Understanding-specific configuration
- Automation Suite Robots-specific configuration
- Monitoring configuration
- Optional: Configuring the proxy server
- Optional: Enabling resilience to zonal failures in a multi-node HA-ready production cluster
- Optional: Passing custom resolv.conf
- Optional: Increasing fault tolerance
- install-uipath.sh parameters
- Adding a dedicated agent node with GPU support
- Adding a dedicated agent Node for Task Mining
- Connecting Task Mining application
- Adding a Dedicated Agent Node for Automation Suite Robots
- Step 15: Configuring the temporary Docker registry for offline installations
- Step 16: Validating the prerequisites for the installation
- Manual: Performing the installation
- Post-installation
- Cluster administration
- Managing products
- Getting Started with the Cluster Administration portal
- Migrating objectstore from persistent volume to raw disks
- Migrating from in-cluster to external High Availability Add-on
- Migrating data between objectstores
- Migrating in-cluster objectstore to external objectstore
- Migrating to an external OCI-compliant registry
- Switching to the secondary cluster manually in an Active/Passive setup
- Disaster Recovery: Performing post-installation operations
- Converting an existing installation to multi-site setup
- Guidelines on upgrading an Active/Passive or Active/Active deployment
- Guidelines on backing up and restoring an Active/Passive or Active/Active deployment
- Redirecting traffic for the unsupported services to the primary cluster
- Monitoring and alerting
- Migration and upgrade
- Step 1: Moving the Identity organization data from standalone to Automation Suite
- Step 2: Restoring the standalone product database
- Step 3: Backing up the platform database in Automation Suite
- Step 4: Merging organizations in Automation Suite
- Step 5: Updating the migrated product connection strings
- Step 6: Migrating standalone Orchestrator
- Step 7: Migrating standalone Insights
- Step 8: Deleting the default tenant
- B) Single tenant migration
- Migrating from Automation Suite on Linux to Automation Suite on EKS/AKS
- Upgrading Automation Suite
- Downloading the installation packages and getting all the files on the first server node
- Retrieving the latest applied configuration from the cluster
- Updating the cluster configuration
- Configuring the OCI-compliant registry for offline installations
- Executing the upgrade
- Performing post-upgrade operations
- Product-specific configuration
- Using the Orchestrator Configurator Tool
- Configuring Orchestrator parameters
- Orchestrator appSettings
- Configuring appSettings
- Configuring the maximum request size
- Overriding cluster-level storage configuration
- Configuring credential stores
- Configuring encryption key per tenant
- Cleaning up the Orchestrator database
- Best practices and maintenance
- Troubleshooting
- How to troubleshoot services during installation
- How to uninstall the cluster
- How to clean up offline artifacts to improve disk space
- How to clear Redis data
- How to enable Istio logging
- How to manually clean up logs
- How to clean up old logs stored in the sf-logs bucket
- How to disable streaming logs for AI Center
- How to debug failed Automation Suite installations
- How to delete images from the old installer after upgrade
- How to disable TX checksum offloading
- How to upgrade from Automation Suite 2022.10.10 and 2022.4.11 to 2023.10.2
- How to manually set the ArgoCD log level to Info
- How to expand AI Center storage
- How to generate the encoded pull_secret_value for external registries
- How to address weak ciphers in TLS 1.2
- Unable to run an offline installation on RHEL 8.4 OS
- Error in downloading the bundle
- Offline installation fails because of missing binary
- Certificate issue in offline installation
- First installation fails during Longhorn setup
- SQL connection string validation error
- Prerequisite check for selinux iscsid module fails
- Azure disk not marked as SSD
- Failure after certificate update
- Antivirus causes installation issues
- Automation Suite not working after OS upgrade
- Automation Suite requires backlog_wait_time to be set to 0
- Volume unable to mount due to not being ready for workloads
- Support bundle log collection failure
- Test Automation SQL connection string is ignored
- Single-node upgrade fails at the fabric stage
- Cluster unhealthy after automated upgrade from 2021.10
- Upgrade fails due to unhealthy Ceph
- RKE2 not getting started due to space issue
- Volume unable to mount and remains in attach/detach loop state
- Upgrade fails due to classic objects in the Orchestrator database
- Ceph cluster found in a degraded state after side-by-side upgrade
- Unhealthy Insights component causes the migration to fail
- Service upgrade fails for Apps
- In-place upgrade timeouts
- Docker registry migration stuck in PVC deletion stage
- AI Center provisioning failure after upgrading to 2023.10
- Upgrade fails in offline environments
- SQL validation fails during upgrade
- snapshot-controller-crds pod in CrashLoopBackOff state after upgrade
- Longhorn REST API endpoint upgrade/reinstall error
- Setting a timeout interval for the management portals
- Authentication not working after migration
- Kinit: Cannot find KDC for realm <AD Domain> while getting initial credentials
- Kinit: Keytab contains no suitable keys for *** while getting initial credentials
- GSSAPI operation failed due to invalid status code
- Alarm received for failed Kerberos-tgt-update job
- SSPI provider: Server not found in Kerberos database
- Login failed for AD user due to disabled account
- ArgoCD login failed
- Update the underlying directory connections
- Failure to get the sandbox image
- Pods not showing in ArgoCD UI
- Redis probe failure
- RKE2 server fails to start
- Secret not found in UiPath namespace
- ArgoCD goes into progressing state after first installation
- MongoDB pods in CrashLoopBackOff or pending PVC provisioning after deletion
- Unhealthy services after cluster restore or rollback
- Pods stuck in Init:0/X
- Missing Ceph-rook metrics from monitoring dashboards
- Pods cannot communicate with FQDN in a proxy environment
- Running High Availability with Process Mining
- Process Mining ingestion failed when logged in using Kerberos
- After Disaster Recovery Dapr is not working properly for Process Mining and Task Mining
- Unable to connect to AutomationSuite_ProcessMining_Warehouse database using a pyodbc format connection string
- Airflow installation fails with sqlalchemy.exc.ArgumentError: Could not parse rfc1738 URL from string ''
- How to add an IP table rule to use SQL Server port 1433
- Running the diagnostics tool
- Using the Automation Suite Support Bundle Tool
- Exploring Logs
Step 1: Preparing the Azure deployment
The deployment requires access to an Azure subscription and a Resource Group with the RBAC role Owner. The Owner role is needed to create a user-assigned Managed Identity with the Contributor role assigned at the Resource Group scope. The Managed Identity is needed for managing the VMs (perform scale-in and scale-out actions, apply instance protection, update the OS).
You can check your role assignment by going through the following:
Resource Group → Access Control (IAM) → Check Access → View My Access
The deployment provisions a number of Standard_D (general purpose), Standard_F and/or Standard_NC (with GPU) VMs. The Azure subscription has a quota on the number of cores that can be provisioned for the VM family.
Some of the deployed VMs must be provisioned with Premium SSDs and, depending on the configuration, Ultra SSDs. Make sure these SSDs are available and are not blocked by any policy.
We use SQL elastic pools to deploy the databases. Make sure that SQL elastic pools are not blocked by any policy.
To check the subscription quota, go to Usage + quotas in the Azure portal.
As part of the installation process, we add instance protection from scale set operations to all nodes of the Server Scales Set. Since these operations are performed from Azure, without the server context, cluster malfunction is prevented. We provide runbooks for cluster management operations. For more about Scale Set Instance Protection, see Azure documentation.
We provide instance termination support for Agent Virtual Machine Instances. This means that when an Agent Virtual Machine Instance is terminated, we cordon, drain, and delete that node from the Automation Suite cluster.
We run a script on each Agent Virtual Machine Instance that is pooling the Instance Metadata Service for Termination events. Whenever receiving an event, we trigger a cordon and a drain command on the respective node, and a server also runs a delete node command for that specific node.
logs
container. Each log file contains the name of the node and has the -termination.log
suffix.
Make sure that the VM SKUs are available for the region in which you deploy.
You can check the availability at: Azure Products by Region.
.crt
certificates are Base64-encoded before providing them.
.pfx
certificate (server certificate). You can then use these strings when filling in the template parameters. You can run this
bash script on a Windows machine using Windows Subsystem for Linux. It uses openssl
to convert the certificates. Keep in mind that the server certificate (the .pfx
) should meet somerequirements.
.pfx
cert password:
pfxFile=<path of the pfx file>
# Key
openssl pkcs12 -in $pfxFile -nocerts -out serverCertKeyEncrypted.key
openssl rsa -in serverCertKeyEncrypted.key -out serverCertKeyDecrypted.key
# Server cert
openssl pkcs12 -in $pfxFile -clcerts -nokeys -out serverCert.crt
# CA Bundle:
openssl pkcs12 -in $pfxFile -cacerts -nokeys -chain | sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' > caBundle.crt
# Converting to base64 and removing newlines
cat serverCertKeyDecrypted.key | base64 | tr -d '\n' > base64CertKey
cat serverCert.crt | base64 | tr -d '\n' > base64Cert
cat caBundle.crt | base64 | tr -d '\n' > base64CABundle
pfxFile=<path of the pfx file>
# Key
openssl pkcs12 -in $pfxFile -nocerts -out serverCertKeyEncrypted.key
openssl rsa -in serverCertKeyEncrypted.key -out serverCertKeyDecrypted.key
# Server cert
openssl pkcs12 -in $pfxFile -clcerts -nokeys -out serverCert.crt
# CA Bundle:
openssl pkcs12 -in $pfxFile -cacerts -nokeys -chain | sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' > caBundle.crt
# Converting to base64 and removing newlines
cat serverCertKeyDecrypted.key | base64 | tr -d '\n' > base64CertKey
cat serverCert.crt | base64 | tr -d '\n' > base64Cert
cat caBundle.crt | base64 | tr -d '\n' > base64CABundle
Connect AiCenter to an external Orchestrator
to true
and provide certificates for Orchestrator and Identity to the parameters listed in Deploying Automation Suite to Azure. For details on how to obtain the certificates, see Chain certificates.
To encode the certificates in base64 format, run the following commands:
cat orchestrator.cer | base64 | tr -d '\n' > orchestratorCert
cat identity.cer | base64 | tr -d '\n' > identityCert
cat orchestrator.cer | base64 | tr -d '\n' > orchestratorCert
cat identity.cer | base64 | tr -d '\n' > identityCert
To register AI Center to the external Orchestrator, you must run the RegisterAiCenterExternalOrchestrator runbook.
By default, the templates deploy the VMs across as many Azure Availability Zones as possible to enable the resilience to zonal failures in a multi-node HA-ready production cluster.
Not all Azure Regions support Availability Zones. See Azure Geograpies for details.
VM SKUs have additional Availability Zones restrictions that you can check using the CLI cmdlet. See Get-AzComputeResourceSku for details.
The cluster is considered resilient to zonal failures if the servers are spread across three Azure Availability Zones. If the Azure region does not support Availability Zones for the type of VM selected for servers, the deployment will continue without zone resilience.
The template provisions an Azure Load Balancer with a public IP and a DNS label to access the services.
<dnsName>.<regionName>.cloudapp.azure.com
.
Azure-provided
or 168.63.129.16
.
If you want to access the cluster over the internet, you can check out Step 3: Post-deployment steps.
The template allows you to deploy the nodes in an existing Virtual Network. However, the Virtual Network must have a subnet that meets the following requirements:
- has enough free address space to accommodate all the nodes and the internal load balancer;
- outbound connectivity; preferably configured through a NAT gateway as per Microsoft recommendation;
- allows HTTPS traffic on port
443
; - Optional: has a service endpoint configured for
Microsoft.Storage
. This is needed if you enable the backup at deployment time.
When deploying into an existing Virtual Network, you must have the Owner RBAC role on it to create a Contributor role assignment at its scope. This is needed for the Instance Refresh operation when scaling out.
# of server nodes
x 512GiB) used as an NFS share and configuring the backup for the cluster. By default, the backup interval is set to 90 minutes,
and the retention interval is 72 hours. You can change the backup and retention intervals post-deployment. For details, see
BackupCluster.