Managing the Certificates

Important: The installation process generates self-signed certificates on your behalf. These certificates will expire in 90 days, and you must replace them with certificates signed by a trusted Certificate Authority (CA) as soon as installation completes. If you do not update the certificates, the installation will stop working in 90 days.

The installation bundle provides a cluster management tool that enables you to update certificates post-installation. To access the tool, navigate to the location of the installer bundle:

cd /opt/UiPathAutomationSuite/

Generating a Certificate Signing Request (CSR) and a Private Key

To generate the CSR and the private key, run the following command:

# copy the machine openssl configuration locally
cp /etc/pki/tls/openssl.cnf ./openssl.tmp.cnf

# Replace the [AUTOMATION_SUITE_FQDN] value. For example, "automationsuite.corp.com"
AS_FQDN=[AUTOMATION_SUITE_FQDN]
cat >> ./openssl.tmp.cnf <<EOF
[SAN]
subjectAltName=DNS:$AS_FQDN,DNS:alm.$AS_FQDN,DNS:monitoring.$AS_FQDN,DNS:registry.$AS_FQDN,DNS:objectstore.$AS_FQDN,DNS:insights.$AS_FQDN
EOF

# create the certificate request
openssl req -new -sha256 -newkey rsa:2048 -nodes -keyout server.key -subj "/C=xx/ST=xx/O=xx/OU=xx/CN=$AS_FQDN" -reqexts SAN -config openssl.tmp.cnf -out ${AS_FQDN}.csr

Your IT team uses the obtained values to generate a signed certificate. The generated private key remains local.

Managing server certificates

To view more information about server certificates, run the following command:

sudo ./configureUiPathAS.sh tls-cert --help

Output:

************************************************************************************

Manage cluster tls and server certificate

Usage:
  configureUiPathAS.sh tls-cert [command]
  configureUiPathAS.sh tls-cert [flags]

Available Commands:
  update                              Update the tls / server certificate

  get                                 Get the tls / server certificate

Flags:
  -h|--help                           Display help

************************************************************************************

The following sections describe the operations you can perform using the ./configureUiPathAS.sh tls-cert command.

Updating the server certificate

Online installation: How to find the server certificate

Certificates are stored as a secret at the Istio level. You can find certificates under the istio-ingressgateway-certs name in the istio-system namespace.

See the certificate files in the following list:

• Server TLS certificate is stored as tls.crt
• Server TLS private key as tls.key
• CA bundle is stored as ca.crt

You can verify the secrets using the following command :

kubectl -n istio-system get secrets istio-ingressgateway-certs -o yaml

Certificates are also stored in the UiPath namespace. This is applicable to every UiPath® product that needs certificate information to trust incoming calls. For details, see Understanding the container architecture related to certificates.

Offline installation: How to find the server certificate

In addition to the certificates required by online deployment, an offline deployment has two additional locations that use the same rootCA.crt and tls.crt: ArgoCD and Docker Registry. Certificates are then stored in both Docker and ArgoCD namespaces.

You can verify the secrets using the following command :

# For docker registry
kubectl -n docker-registry get secrets docker-registry-tls -o yaml
# For Argocd
argocd login alm.cluster_fqnd --username argocd_username --password argocd_password
argocd cert list --cert-type https

How to update server certificates

Important: You must decrypt the certificate key before updating the server certificate. Skipping the decryption step would result in an error.

To decrypt the certificate key, run the following command:

# replace /path/to/encrypted/cert/key to absolute file path of key
# replace /path/to/decrypt/cert/key to store decrypt key
# Once prompted, please entry the passphrase or password to decrypt the key

openssl rsa -in /path/to/encrypted/cert/key -out /path/to/decrypt/cert/key

Run the configureUiPathAS.sh script to update the certificate as shown below. You need the path to each of the three certificate files. All the certificate file should be in PEM format.

• Certificate Authority Bundle - This bundle should contain only the chain certificates used to sign the TLS server certificate. The chain limit is up to nine certificates.
• Server Certificate - Public server certificate

• Private key - Private key for server certificate

  sudo ./configureUiPathAS.sh tls-cert update --ca-cert-file /path/to/cacert --tls-cert-file /path/to/tlscert --tls-key-file /path/to/tlskey

Below files will be stored in in the /directory/path/to/store/certificate location.

Accessing the TLS certificate

To print out the certificate files, run the following command, specifying the directory where certificates are stored.

sudo ./configureUiPathAS.sh tls-cert get --outpath /directory/path/to/store/certificate

Adding the CA certificate to the host trust store

You are responsible for making sure the generated certificates are trusted.

To add the certificate to the host VM trust store, run the following commands on all the nodes in the cluster:

# 1. Copy the certificate file to the /usr/share/pki/ca-trust-source/anchors/ or the /etc/pki/ca-trust/source/anchors/ directory
cp /path/to/the/ca-cert /usr/share/pki/ca-trust-source/anchors/

# 2. Update the trust store configuration
update-ca-trust

Managing additional CA certificates

To view more information about additional CA certificates, run the following command:

./configureUiPathAS.sh additional-ca-certs --help

Output:

***************************************************************************************

Manage additional CA certificates, this can be used to add sql server CA

Usage:
  configureUiPathAS.sh additional-ca-certs [command]
  configureUiPathAS.sh additional-ca-certs [flags]

Available Commands:
  update                              Update the additional trusted CA certificates.

  get                                 Get the additional trusted CA certificates

Flags:
  -h|--help                           Display help

***************************************************************************************

The following sections describe the operations you can perform using the ./configureUiPathAS.sh additional-ca-certs command.

Updating the CA certificates

This command helps you update or replace the existing configured CA certificates.

./configureUiPathAS.sh additional-ca-certs update --ca-cert-file /path/to/ca/certs

Note:

The command above adds a new certificate to the list of existing certificates. if you want to replace all the previously configured certificates, make sure to append --replace at the end.

The CA Certificate bundle file should be a valid .pem format and can have more than one certificate present in it.

Accessing the CA certificates

To download the already configured CA certificates, run the following command:

./configureUiPathAS.sh additional-ca-certs get --outpath /path/to/download/certs

Adding the CA certificate to the host trust store

You are responsible for making sure the generated certificates are trusted.

To add the certificate to the host VM trust store, run the following commands on all the nodes in the cluster:

# 1. Copy the certificate file to the /usr/share/pki/ca-trust-source/anchors/ or the /etc/pki/ca-trust/source/anchors/ directory
cp /path/to/the/ca-cert /usr/share/pki/ca-trust-source/anchors/

# 2. Update the trust store configuration
update-ca-trust

Managing identity token-signing certificates

To view more information about identity token-signing certificates, run the following command:

sudo ./configureUiPathAS.sh identity token-cert --help

Output:

************************************************************************************

Manage Identity token signing certificate

Usage:
  configureUiPathAS.sh identity token-cert [command]
  configureUiPathAS.sh identity token-cert [flags]

Available Commands:
  update                              Update secondary certificate to signing
                                        the authentication token
  rotate                              Switch secondary certificate as a primary
                                        token signing certificate
  get                                 Get token signing certificate

Flags:
  -h|--help                           Display help

************************************************************************************

The following section provide details on the operations you can perform using the./configureUiPathAS.sh identity token-cert command.

Updating the certificate

To upload the new certificate to sign the token, run the following command:

Note:

The following command does not replace the existing token signing certificate.

Make sure the certificate you provide is in .pem format.

sudo ./configureUiPathAS.sh identity token-cert update --cert-file-path /path/to/cert --cert-key-file-path /path/to/certkey

Rotating the certificate

To rotate or replace the old certificate with the new one, run the following command:

sudo ./configureUiPathAS.sh identity token-cert rotate

Note:

There should be the lead time of about 24 - 48 hours between certificate update and rotate.

We need this lead time to keep supporting the authentication for cached token signed by old certificate.

If you rotate the certificate too soon before the expiry of cache token can result in downtime. And you may have to restart all your robots.

Emergency certificate rotation

Important: The following procedure is for emergencies only. You should rotate certificates ahead of their expiration dates

To perform an emergency certificate update, take the following steps:

1. Obtain a new certificate or create a self-signed one and copy it to the cluster server node used to execute the next rotation steps. To create a new self-signed certificate, run the following command:

   openssl req -x509 -sha256 -nodes -days 365 -newkey rsa:2048 -keyout identityserver.key -out identityserver.crt
   openssl pkcs12 -export -out identityserver.pfx -inkey identityserver.key -in identityserver.crt

2. If IdentityServer1.pfx is expired, rotate and update the certificate. For instructions, see Rotating the certificate.
3. If IdentityServer2.pfx is expired, update the certificate.
4. If both certificates are expired, update, rotate and update again.
5. Restart all deployments.
6. Clear all browser caches. If you run in Incognito or Private mode, you can skip this step.
7. For Firefox, press CTRL+SHIFT+DEL, select Cache, and click OK.
   
   

   

   
   
8. For Chrome, press CTRL+SHIFT+DEL, select Cached images and files, and click Clear data.
   
   

   

   
   

Accessing the certificate

Run the following command to download the current token signing certificate:

sudo ./configureUiPathAS.sh identity token-cert get --outpath /directory/path/to/store/certificate

Managing RKE2 certificates

By default, RKE2 certificates expire in 12 months. In the 90 days prior to their expiration date, certificates are rotated when you restart RKE2.

For more details, refer to RKE2 - Advanced Options - Certificate rotation.

Checking the RKE2 certificate expiration date

To check the expiration date of the RKE2 certificate, run the following command on any of the nodes:

if [[ -d "/var/lib/rancher/rke2/server/tls" ]]; then
  dir="/var/lib/rancher/rke2/server/tls"
elif [[ -d "/var/lib/rancher/rke2/agent/tls" ]]; then
  dir="/var/lib/rancher/rke2/agent/tls"
else
dir="/var/lib/rancher/rke2/agent/"
fi
# Loop through each .crt file in the directory
for file in "$dir"/*.crt; do
# Extract the expiry date from the certificate
expiry=$(openssl x509 -enddate -noout -in "$file" | cut -d= -f 2-)
# Get the file name without the path
filename=$(basename "$file")
# Print the filename and expiry date in a pretty format
printf "%-30s %s\n" "$filename:" "$expiry"
done

The output you get should be similar to the one shown in the following image:

Rotating the RKE2 certificate

By default, RKE2 certificates expire in 12 months. In the 90 days prior to their expiration date, certificates are rotated when you restart RKE2. However, if the validity of the certificates exceeds the 90-day period, you must manually rotate the certificates by following the steps mentioned in RKE2 - Advanced Options - Certificate rotation.

If you want to customize the expiration period of RKE2 certificates to meet particular requirements, you can do so before restarting the RKE2 services for both server and agent nodes.

To rotate the RKE2 certificates, you must first execute a series of action on the server nodes, then proceed with some steps on the agent nodes.

Take the following steps on the server nodes:

1. Stop the RKE2 server:

   systemctl stop rke2-server.service

2. Clear any remaining RKE2 processes:

   rke2-killall.sh

3. Delete the dynamic-cert.json file located at /var/lib/rancher/rke2/server/tls/.

4. To customize the expiration period of the RKE2 certificates, use the following command. Be aware that this example sets the validity period to 1000 days, but you can change this value based on your requirements.

   SERVICE_NAME="rke2-server.service"
   conf_file_path="/etc/systemd/system/${SERVICE_NAME}.d/cert.conf"
   mkdir -p /etc/systemd/system/"${SERVICE_NAME}".d/
   
   cat > "$conf_file_path" <<EOF
   [Service]
   Environment="CATTLE_NEW_SIGNED_CERT_EXPIRATION_DAYS=1000"
   EOF
   
   systemctl daemon-reload

5. Restart the RKE2 server:

   systemctl start rke2-server.service

   Note: If the cluster has more than one server node, steps 1-4 might not wholly execute as etcd may be unable to complete the leader election. Should this happen, repeat steps from 1-4 on other server nodes.
6. Delete the rke2-serving secret from the kube-system namespace:

   kubectl delete secret -n kube-system rke2-serving

   Note:

   In a multi-node deployment, you may not be able to run the kubectl commands until you have completed the first four operations on the necessary number of server nodes. This is to fulfill the etcd quorum requirement. You can remove the rke2-serving secret immediately after the RKE2 server starts up.

Once etcd attains quorum, the RKE2 server can start the rest of the control plane pods. You should then see the successful execution of the kubectl get nodes command. When your server nodes are ready, you can proceed to the agent nodes to regenerate the certificates.

Take the following steps on the agent nodes:

1. Stop the RKE2 server:

   systemctl stop rke2-agent.service

2. Clear any remaining RKE2 processes:

   rke2-killall.sh

3. To customize the expiration period of the RKE2 certificates, use the following command. Be aware that this example sets the validity period to 1000 days, but you can change this value based on your requirements.

   SERVICE_NAME="rke2-agent.service"
   conf_file_path="/etc/systemd/system/${SERVICE_NAME}.d/cert.conf"
   mkdir -p /etc/systemd/system/"${SERVICE_NAME}".d/
   
   cat > "$conf_file_path" <<EOF
   [Service]
   Environment="CATTLE_NEW_SIGNED_CERT_EXPIRATION_DAYS=1000"
   EOF
   
   systemctl daemon-reload

4. Restart the RKE2 server:

   systemctl start rke2-agent.service