Offline Multi-node HA-ready Production Mode

Preparation

1. Identify any server (not agent) that meets the disk requirements for an offline installation. Referenced as a primary server throughout this document.

   If you are using a self-signed certificate, run the following command:

   ### Please replace /path/to/cert with path to location where you want to store certificates.
   sudo ./configureUiPathAS.sh tls-cert get --outpath /path/to/cert
   
   ### Now copy the ca.crt file generated in above location to trust store location
   sudo cp --remove-destination /part/to/cert/ca.crt /etc/pki/ca-trust/source/anchors/
   
   ### Update the trust store
   sudo update-ca-trust

2. Download the full offline bundle (sf.tar.gz) on the selected server.
3. Download the infra-only offline bundle (sf-infra.tar.gz) on all the other nodes.
4. Download and unzip the new installer (installer.zip) on all the nodes.
   Note: Give proper permissions to the folder by running sudo chmod 755 -R <installer-folder>.
5. Make the original cluster_config.json available on the primary server.
6. Generate the new cluster_config.json file as follows:

   • If you have the old ``cluster_config.json` file, use the following command to generate the configuration file from the cluster:

     cd /path/to/new-installer ./configureUiPathAS.sh config get -i /path/to/old/cluster_config.json -o /path/to/store/generated/cluster_config.json

   • If you do not have the old cluster config file, run the following command:

     cd /path/to/new-installer ./configureUiPathAS.sh config get -o /path/to/store/generated/cluster_config.json

     Note: See Advanced installation experience to fill up the remaining parameters.
7. Copy this cluster_config.json to the installer folder on all nodes.

Execution

Maintenance and Backup

1. Make sure you have enabled the backup on the cluster. For details, see Backing up and restoring the cluster.
2. Connect to one of the server nodes via SSH.
3. Verify that all desired volumes have backups in the cluster by running the following command:

   /path/to/new-installer/configureUiPathAS.sh verify-volumes-backup

   Note: The backup might take some time, so wait for approximately 15-20 minutes, and then verify the volumes backup again.
4. To verify if Automation Suite is healthy, run:

   kubectl get applications -n argocd

5. Put the cluster in maintenance mode as follows:
   1. Execute the following command:

      /path/to/new-installer/configureUiPathAS.sh enable-maintenance-mode

   2. Verify that the cluster is in maintenance mode by running the following command:

      /path/to/new-installer/configureUiPathAS.sh is-maintenance-enabled

6. Create the SQL database backup.

Upgrade Infrastructure on Servers

Note: Upgrading the infrastructure on servers and agents simultaneously is not supported and will result in an error. Make sure to carry out these steps successively.

1. Connect to each server via SSH.
2. Become root by running sudo su -.
3. Execute the following command on all server nodes:

   /path/to/new-installer/install-uipath.sh --upgrade -k -i /path/to/cluster_config.json --offline-bundle "/path/to/sf-infra.tar.gz" --offline-tmp-folder /uipath/tmp --install-offline-prereqs --accept-license-agreement -o /path/to/output.json

   Note: This command also creates a backup of the cluster state and pauses all other scheduled backups.

Upgrade Infrastructure on Agents

Note: Upgrading the infrastructure on servers and agents simultaneously is not supported and will result in an error. Make sure to carry out these steps successively.

1. Connect to each server via SSH.
2. Become root by running sudo su -.
3. Execute the following command:

   /path/to/new-installer/install-uipath.sh --upgrade -k -i /path/to/cluster_config.json --offline-bundle "/path/to/sf-infra.tar.gz" --offline-tmp-folder /uipath/tmp --install-offline-prereqs --accept-license-agreement -o /path/to/output.json

Execute the Rest of the Upgrade on the Primary Server

1. Connect to primary server via SSH.
2. Become root by running sudo su -.
3. Execute the following command:

   /path/to/new-installer/install-uipath.sh --upgrade -f -s -i /path/to/cluster_config.json --offline-bundle "/path/to/sf.tar.gz" --offline-tmp-folder /uipath/tmp --install-offline-prereqs --accept-license-agreement -o /path/to/output.json

   Note: This command disables the maintenance mode that you enabled before the upgrade because all services are required to be up during the upgrade.
4. After the successful upgrade and verification, resume the backup scheduling on the node by running the following command:

   /path/to/new-installer/configureUiPathAS.sh resume-scheduled-backups

Rollback on Error

Preparation

1. Create a separate folder to store the old bundles, and perform the following operations inside that folder.
2. Download and unzip the installer's older version (installer.zip) on all the nodes.
   Note: Give proper permissions to the folder by running sudo chmod 755 -R <installer-folder>.
3. Create restore.json file and copy it to all the nodes. For details, see Backing up and restoring the cluster.
4. Verify that the etcd backup data is present on the primary server at the following location: /mnt/backup/backup/<etcdBackupPath>/<node-name>/snapshots
   • etcdBackupPath - this is the same as the one specified in backup.json while enabling the backup's node-name;
   • node-name - the hostname of the primary server VM.

Cluster Cleanup

1. Copy and run the dedicated script to uninstall everything from that node. Do this for all the nodes. For details, see Troubleshooting.
2. Restore all UiPath databases to the older backup that was created before the upgrade.

Restore Infra on Server Nodes

1. Connect to the primary server (which is the same as the one chosen during upgrade).
2. Restore infra by running the following command:

   /path/to/older-installer/install-uipath.sh -i /path/to/restore.json -o /path/to/output.json -r --accept-license-agreement --install-type online

3. Connect to the rest of the server nodes one by one via SSH.
4. Restore infra on these nodes by running the following command:

   /path/to/older-installer/install-uipath.sh -i /path/to/restore.json -o /path/to/output.json -r -j server --accept-license-agreement --install-type online

   Note: Run this command on server nodes one by one. Executing them in parallel is not supported.

Restore Infra on Agent Nodes

1. Connect to each agent VM via SSH.
2. Restore infra on these nodes by running the following command:

   /path/to/older-installer/install-uipath.sh -i /path/to/restore.json -o /path/to/output.json -r -j agent --accept-license-agreement --install-type online

Restore Volumes Data

1. Connect to the primary server via SSH.
2. Go to the new installer folder.
   Note: The previous infra restore commands were executed using older installer, and the following commands are executed using newer installer bundle.
3. Disable the maintenance mode on the cluster by running the following command:

   /path/to/new-installer/configureUiPathAS.sh disable-maintenance-mode

4. Verify that maintenance mode is disabled by executing the following command:

   /path/to/new-installer/configureUiPathAS.sh is-maintenance-enabled

5. Copy the restore.jsonfile that was used in the infra restore stage to the new installer bundle folder.
6. Restore volumes from the newer installer bundle by executing the following command:

   /path/to/new-installer/install-uipath.sh -i /path/to/new-installer/restore.json -o /path/to/new-installer/output.json -r --volume-restore --accept-license-agreement --install-type online

7. Once the restore is completed, verify if everything is restored and working properly.
8. During the upgrade, scheduled backups were disabled on the primary node. To enable them again, run the following command:

   /path/to/new-installer/configureUiPathAS.sh resume-scheduled-backups