Online Multi-node HA-ready Production Mode

Preparation

1. Identify any server (not agent) that meets the disk requirements for an online installation. Referenced as primary server throughout this document. If you are using a self-signed certificate, run the following command:

   ### Please replace /path/to/cert with path to location where you want to store certificates.
   sudo ./configureUiPathAS.sh tls-cert get --outpath /path/to/cert
   
   ### Now copy the ca.crt file generated in above location to trust store location
   sudo cp --remove-destination /part/to/cert/ca.crt /etc/pki/ca-trust/source/anchors/
   
   ### Update the trust store
   sudo update-ca-trust

2. Download and unzip the new installer (installer.zip) on all the nodes.
   Note: Give proper permissions to the folder by running sudo chmod 755 -R <installer-folder>.
3. Make the original cluster_config.json file available on the primary server.
4. Generate the new cluster_config.json file as follows:
   • If you have the old cluster_config.json file, use the following command to generate the configuration file from the cluster:

     cd /path/to/new-installer ./configureUiPathAS.sh config get -i /path/to/old/cluster_config.json -o /path/to/store/generated/cluster_config.json

   • If you do not have the old configuration file, run the following command:

     cd /path/to/new-installer ./configureUiPathAS.sh config get -o /path/to/store/generated/cluster_config.json

     Note: See Advanced installation experience to fill up the remaining parameters.
5. Copy this cluster_config.json to the installer folder on all nodes.

Execution

Maintenance and Backup

1. Make sure you have enabled the backup on the cluster. For details, see Enabling the cluster backup.
2. Connect to one of the server nodes via SSH.
3. Verify that all desired volumes have backups in the cluster by running /path/to/new-installer/configureUiPathAS.sh verify-volumes-backup.
   Note: The backup might take some time, so wait for approximately 15-20 minutes, and then verify the volumes backup again.
4. To verify if Automation Suite is healthy, run:

   kubectl get applications -n argocd

5. Put the cluster in maintenance mode as follows:
   1. Execute the following command:

      /path/to/new-installer/configureUiPathAS.sh enable-maintenance-mode

   2. Verify that the cluster is in maintenance mode by running the following command:

      /path/to/new-installer/configureUiPathAS.sh is-maintenance-enabled

6. Create the SQL database backup.

Upgrade Infrastructure on Servers

1. Connect to each server via SSH.
2. Become root by running sudo su -.
3. Execute the following command on all servers:

   /path/to/new-installer/install-uipath.sh --upgrade -k -i /path/to/cluster_config.json --accept-license-agreement -o /path/to/output.json

As part of this command, we take the backup of the Cluster state as well and pause all the further scheduled backups.

Note: This command also creates a backup of the cluster state and pauses all other scheduled backups.

Upgrade Infrastructure on Agents

1. Connect to each server via SSH.
2. Become root by running sudo su -.
3. Execute the following command:

   /path/to/new-installer/install-uipath.sh --upgrade -k -i /path/to/cluster_config.json --accept-license-agreement -o /path/to/output.json

Execute Rest of the Upgrade on the Primary Server

1. Connect to primary server via SSH.
2. Become root by running sudo su -.
3. Execute the following command:

   /path/to/new-installer/install-uipath.sh --upgrade -f -s -i /path/to/cluster_config.json --accept-license-agreement -o /path/to/output.json

   Note: This command disables the maintenance mode that you enabled before the upgrade because all services are required to be up during the upgrade.
4. After the successful upgrade and verification, resume the backup scheduling on the node by running the following command:

   /path/to/new-installer/configureUiPathAS.sh resume-scheduled-backups

Rollback on Error

Preparation

1. Create a separate folder to store the old bundles, and perform the following operations inside that folder.
2. Download and unzip the installer's old version (installer.zip) on all the nodes.
   Note: Give proper permissions to the folder by running sudo chmod 755 -R <installer-folder>.
3. Create the restore.json file and copy it to all the nodes. For details, see Backing up and restoring the cluster.
4. Verify that the etcd backup data is present on the primary node at the following location: /mnt/backup/backup/<etcdBackupPath>/<node-name>/snapshots
   • etcdBackupPath - this is the same as the one specified in backup.json while enabling the backup's node-name;
   • node-name - the hostname of the primary server VM.

Cluster Cleanup

1. Copy and run the dedicated script to uninstall everything from that node. Do this for all the nodes. For details, see Troubleshooting.
2. Restore all UiPath databases to the older backup that was created before the upgrade.

Restore Infra on Server Nodes

1. Connect to the primary server. This should be the same server node you selected during the upgrade.
2. Restore infra by running the following command:

   /path/to/older-installer/install-uipath.sh -i /path/to/restore.json -o /path/to/output.json -r --accept-license-agreement --install-type online

3. Connect to the rest of the server nodes one by one via SSH.
4. Restore infra on these nodes by running the following command on the server nodes one by one. Executing them in parallel is not supported.

   /path/to/older-installer/install-uipath.sh -i /path/to/restore.json -o /path/to/output.json -r -j server --accept-license-agreement --install-type online

Restore Infra on Agent Nodes

1. Connect to each agent VM via SSH.
2. Restore infra on these nodes by running the following command:

   /path/to/older-installer/install-uipath.sh -i /path/to/restore.json -o /path/to/output.json -r -j agent --accept-license-agreement --install-type online

Restore Volumes Data

1. Connect to the primary server via SSH.
2. Go to the newer installer folder.
   Note: The previous infra restore commands were executed using the old installer, and the following commands are executed using the new installer bundle.
3. Disable the maintenance mode on the cluster by running the following command:

   /path/to/new-installer/configureUiPathAS.sh disable-maintenance-mode

4. Verify that maintenance mode is disabled by running the following command:

   /path/to/new-installer/configureUiPathAS.sh is-maintenance-enabled

5. Copy the restore.json file that was used in the infra restore stage to the new installer bundle folder.
6. Run volumes restore from the new installer bundle by executing the following command:

   /path/to/new-installer/install-uipath.sh -i /path/to/new-installer/restore.json -o /path/to/new-installer/output.json -r --volume-restore --accept-license-agreement --install-type online

7. Once the restore is completed, verify if everything is restored and working properly.
8. During the upgrade, scheduled backups were disabled on the primary node. To enable them again, run the following command:

   /path/to/new-installer/configureUiPathAS.sh resume-scheduled-backups