Automated: Offline upgrade

Ansible is used to automate the upgrade of the entire Automation Suite cluster, by performing some operations on one machine in the cluster.

In the following illustration, the Server 1 node (or Ansible host node) uses Ansible to orchestrate the upgrade of entire Automation Suite cluster.

Preparation

Important:

Before upgrading, make sure you have prepared your environment for using the uipathctl.sh script. For details, see Using uipathctl.sh.

Make sure you have 10GiB of free space in the /opt/UiPathAutomationSuite folder on all the nodes. If you do not have enough space, you can either increase the capacity of this folder or remove all the previous installer files except for cluster_config.json. You can always download the previous installer again.

To verify available space, run the following command: df -h /opt/UiPathAutomationSuite.

To prepare for the upgrade, take the following steps:

1. Log into the server node that you previously identified as the Ansible host node via SSH. Make sure you have selected the primary server node with the UiPath bundle additional disk attached at /uipath as your Ansible host node.
2. Become root user:

   sudo su -

3. Create an installation folder under the /opt/UiPathAutomationSuite directory:

   # Please replace {version} with installer version
   mkdir -p /opt/UiPathAutomationSuite/{version}/installer

4. Download as-installer.zip and copy it to /opt/UiPathAutomationSuite/{version}/installer. For download instructions, see as-installer.zip.

   cp /path/to/as-installer.zip /opt/UiPathAutomationSuite/{version}/installer

5. Unzip as-installer.zip:

   cd /opt/UiPathAutomationSuite/{version}/installer
   unzip as-installer.zip -d .

6. Download as-infra.tar.gz and copy it to /uipath/{version}. For download instructions, see as-infra.tar.gz.

   cp /path/to/as-infra.tar.gz /uipath/{version}

7. Download as.tar.gz and copy it to /uipath/{version}. For download instructions, see as.tar.gz.

   cp /path/to/as.tar.gz /uipath/{version}

8. Provide the necessary permissions to the installer:

   chmod 755 -R /opt/UiPathAutomationSuite/{version}/installer

9. Generate the latest cluster_config.json file on the Ansible host node.
   Note: For details on how to configure the cluster_config.json parameters, see Advanced installation experience.

   • If you have the old cluster_config.json, generate the configuration file from the cluster:

     cd /opt/UiPathAutomationSuite/{version}/installer
     
     ./configureUiPathAS.sh config get -i /path/to/old/cluster_config.json -o ./cluster_config.json

   • If you do not have the old cluster_config.json file, generate any override of any default values, which you may have done at the time of the previous version installation:

     cd /opt/UiPathAutomationSuite/{version}/installer
     
     ./configureUiPathAS.sh config get -o ./cluster_config.json

   • If you upgrade from Automation Suite 2022.10 to Automation Suite 2023.4 or later, and Orchestrator is enabled on both the old and new versions, you must set the orchestrator.block_classic_executions flag to true in the cluster_config.json file. Using the flag shows that you agree with blocking classic folders executions. Not using the flag causes the upgrade operation to fail. For a configuration example, see Orchestrator-specific configuration.
   • If you upgrade Automation Suite from a previous version and now enable Process Mining as well, you must update the sql_connection_string_template_sqlalchemy_pyodbc. Below is an example.

     "sql_connection_string_template_sqlalchemy_pyodbc": "mssql+pyodbc://[e-mail]:[e-mail]:1433/DB_NAME_PLACEHOLDER?driver=ODBC+Driver+17+for+SQL+Server"

   • If you upgrade Automation Suite from a version previous to 2023.4 and you have Apps enabled, you must create the SQL database for Apps, and then update the cluster_config.json file with the following configuration:

     "apps": {
     "enabled": "true" // This will already be true in existing cluster_config.json if apps is enabled
     "sql_connection_str": "" // Provide ODBC database connection string if apps is enabled
     }

     Note:

     Use the updated cluster_config.json file in the next upgrading steps.

   See also Manual: Advanced installation experience.

Execution

Important:

If any errors or issues occur during or following the upgrade, you can rollback to the previous version, provided that you previously configured a backup.

For instructions, see Rollback on error.

Hydrating Docker registry

To minimize the required downtime, hydrate docker-registry with container images for the new version of products before the upgrade.

Important:

• Make sure not to hydrate containerd and/or other infra images more that 10 hours before the upgrade.
• If you upgrade Document Understanding and Computer Vision, you must download the base images that match your upgraded Automation Suite version (use the base image that corresponds with your ML Package version) and hydrate them to your docker registry prior to upgrade to avoid disruption to your running DU skills. To download the Document Understanding and Computer Vision installation packages, see Downloading the installation packages.

To hydrate docker-registry, run the following command on the first server node:

cd /opt/UiPathAutomationSuite/{version}/installer

./configureUiPathAS.sh registry upload --offline-bundle /uipath/{version}/as.tar.gz --offline-tmp-folder /uipath/tmp

Configuring the backup

To configure the backup, take the following steps:

1. Make sure you have enabled the backup on the cluster. You must create the backup using the same version of the installer as the one you used for the current deployment. For instructions, see the backup and restore documentation corresponding to the Automation Suite version from which you plan to upgrade. For instance, if you plan to upgrade from Automation Suite 2021.10, follow the instructions in the 2021.10 guide.
2. Log into the server node that you previously identified as the Ansible host node via SSH.
3. Verify that all desired volumes have backups in the cluster:

   /path/to/old-installer/configureUiPathAS.sh verify-volumes-backup

   Note: The backup might take some time, so wait for approximately 15-20 minutes, and then verify the volumes backup again.

Once the backup is created, continue with the following steps.

Putting the cluster in maintenance mode

Putting the cluster in maintenance mode will shut down the ingress controller and all the UiPath services, blocking all the incoming traffic to the Automation Suite cluster.

Important: This operation will cause downtime, and your business automation will be suspended during the entire upgrade process.

1. To put the cluster in maintenance mode, run:

   cd /opt/UiPathAutomationSuite/{version}/installer
   ./configureUiPathAS.sh enable-maintenance-mode

2. To verify that the cluster is in maintenance mode, run:

   ./configureUiPathAS.sh is-maintenance-enabled

   Important: Create a backup of your SQL Server after enabling maintenance mode so that no further transactions are recorded in your SQL database.

Upgrading the cluster

To upgrade the Automation Suite cluster, take the following steps:

1. Connect to the Ansible host node via SSH and become root user.
2. To perform the upgrade steps on all the nodes, run the following command with the basic parameters. For more granular control, see the parameters section.
   Note: Make sure you have updated cluster_config.json generated as described in the Preparation steps.

   # Please replace {version} with installer version, in all the below commands
   cd /opt/UiPathAutomationSuite/{version}/installer
   
   ./uipathctl.sh upgrade --cluster-config ./cluster_config.json --install-type offline --steps all --offline-infra-bundle /uipath/{version}/as-infra.tar.gz --accept-license-agreement

   Important:

   The upgrade process requires migrating from Canal CNI to Cilium CNI. The automated upgrade performs this step automatically and then restarts all nodes in the cluster to complete the migration.

   Once the nodes are restarted, wait for a few minutes to ensure that all services are online again. You can log into ArgoCD to monitor the status of all applications.

   Important:

   If you get the following error, follow the instructions in Downgrading Ceph from 16.2.6 to 15.2.9.

   Ceph cluster is running on version: 16.2.6 with known bug(https://docs.ceph.com/en/quincy/releases/pacific/#v16-2-6-pacific). Please provide temporary backup directory (via environment variable 'TMP_CEPH_BACKUP_PATH') to downgrade ceph

Basic parameters

Parameter	Value
--install-type	Possible values: online and offline. Since this page provides instructions for an offline environment, choose the offline value.
--steps	Possible values are: all, infra, fabric, and services. Denotes each stage of the installation and upgrade process. all – Recommended: A superset of all the other steps, which upgrades the entire Automation Suite. infra – Upgrades Kubernetes and other infrastructure components on Automation Suite on all the nodes. bundles – Uploads and hydrates the docker registry hosted within the Automation Suite cluster with the fabric and services container images. Important: You must upload and hydrate the Docker registry at the same time as upgrading Kubernetes and the infrastructure components. To execute the two steps simultaneously, run --steps bundles infra. Do not run --steps bundles alone as it will damage the cluster. fabric – Upgrades all fabric and shared components of Automation Suite. services – Upgrades all UiPath product services installed on Automation Suite. If an error occurs during the upgrade, you can use these checkpoints to resume the upgrade operation instead of retrying from start.
--offline-bundle	Path to the as.tar.gz offline bundle, containing the fabric and services container images. It is recommended to store it in /uipath/{version}/as.tar.gz
--offline-infra-bundle	Path to the as-infra.tar.gz offline bundle, containing Kubernetes and infrastructure bundles.
--cluster-config	cluster_config.json file, which defines the parameters, settings, and preferences applied to the UiPath services deployed via Automation Suite.

Advanced parameters

Parameter	Value
--ansible-username	Specify the username to be used for SSH connections to all the nodes. Defaults to the current user. If you use a different username for all the nodes, then instead of using this parameter, set the username for all the nodes in inventory.ini and pass it to the script. Example: While running the script, you are logged in as the myadminuser username. However, if you want to use the testadmin username to connect via SSH, you must provide testadmin to this parameter.
--inventory	Specify an existing Ansible inventory file. If not specified, one will be generated automatically from the existing cluster with the basic configuration. In certain scenarios where you want to have more granular control, you may want to provide your own inventory file. One such example would be if you want to use a different SSH username or SSH key for all the nodes in cluster. For details, see Generating the Ansible inventory.ini file.

Note:

If the upgrade failed for any reason, but the upload to the Docker Registry is successful, no re-upload is required. You can resume the upgrade by directly running the fabric or service installation: ./uipathctl.sh upgrade --install-type offline --steps {fabric | services}.

Once the upgrade is done, maintenance mode will be disabled automatically.

After performing the upgrade, you can take the following additional steps:

• To verify if Automation Suite is healthy, run:

  kubectl get applications -n argocd

• When upgrading from an Automation Suite version prior to 2023.4.0, verify if Apps is running, then remove MongoDB:

  ./configureUiPathAS.sh mongodb uninstall --force

• If you face an error when removing MongoDB with ./configureUiPathAS.sh mongodb uninstall --force command, run the following command:

  kubectl patch application "fabric-installer" -n argocd --type=merge -p '{"spec" : {"syncPolicy" : {"automated" : {"selfHeal": false }}}}'
  
  ./configureUiPathAS.sh mongodb uninstall --force
  
  kubectl patch application "fabric-installer" -n argocd --type=merge -p '{"spec" : {"syncPolicy" : {"automated" : {"selfHeal": true }}}}'

• If you experience issues with image vulnerabilities or storage consumption after performing an upgrade, delete the images from the old installer. For details, see the Troubleshooting section.

• After upgrading to Automation Suite 2023.4.8 or newer, you can clean up the previous version images after the upgrade from the in-cluster Docker registry. For instructions, see How to delete images from the old installer after upgrade.

Enabling the backup post-upgrade

Note: Make sure Automation Suite is up and running and your automation continues as expected before proceeding with the next steps.

Before starting to upgrade the cluster, the upgrade script automatically creates a backup of the cluster and then temporarily disables it. You may want to enable the backup manually once the upgrade is done.

To enable the backup, follow the instructions in described in the Backing up and restoring the cluster documentation.