Using the monitoring stack

The monitoring stack for Automation Suite clusters includes Prometheus, Grafana, and Alertmanager, which are integrated within the Rancher Cluster Explorer UI.

Note:

Node failures might lead to a Kubernetes shutdown, which would disrupt Prometheus alerts. To prevent this, we recommend setting up a separate alert on the RKE2 server.

This page describes a series of monitoring scenarios. For more details, see the official Rancher documentation on using Rancher Monitoring.

Important:

When using collectors to export metrics to third-party tools, enabling application monitoring may disrupt the functionality of Automation Suite.

Accessing the monitoring tools

Overview

The monitoring stack for Automation Suite clusters includes Prometheus, Grafana, Alert Manager, and Longhorn Dashboard.

This page describes a series of monitoring scenarios.

For more details, see the official Rancher documentation on using Rancher Monitoring.

You can access the Automation Suite monitoring tools individually using the following URLs:

Application	Tool	URL	Example
Metrics	Prometheus	https://monitoring.fqdn/metrics	https://monitoring.automationsuite.mycompany.com/metrics
Dashboard	Grafana	https://monitoring.fqdn/dashboard	https://monitoring.automationsuite.mycompany.com/dashboard
Alert Management	Alert Manager	https://monitoring.fqdn/alertmanager	https://monitoring.automationsuite.mycompany.com/alertmanager
Persistent Block Storage	Longhorn dashboard	https://monitoring.fqdn	https://monitoring.automationsutie.mycompany.com

Authentication

To access the monitoring tools for the first time, log in as an admin with the following default credentials:

• Username: admin
• Password: to retrieve the password , run the following command:

  kubectl get secrets/dex-static-credential -n uipath-auth -o "jsonpath={.data['password']}" | base64 -d

To update the default password used for Dex authentication while accessing the monitoring tools, take the following steps:

1. Run the following command by replacing newpassword with your new password:

   password="newpassword"
   password=$(echo -n $password | base64)
   kubectl patch secret dex-static-credential -n uipath-auth --type='json' -p="[{'op': 'replace', 'path': '/data/password', 'value': '$password'}]"

2. Run the following command by replacing <cluster_config.json> with the path to your configuration file:

   /opt/UiPathAutomationSuite/UiPath_Installer/install-uipath.sh -i <cluster_config.json> -f -o output.json --accept-license-agreement

Checking currently firing alerts

To view the alerts, navigate to Prometheus using https://monitoring.fqdn/metrics and click the Alerts tab. Here you can see all the alerts configured in Automation Suite.

To view the active alerts, filter the alert status by clicking the Firing checkbox and the Show annotations checkbox at the top. Here you can see all the alerts that are firing currently and their corresponding messages.

Silencing alerts

If alerts are too noisy, you can silence them. To do that, take the following steps:

1. Click the Alertmanager tile in the upper left corner of the Monitoring dashboard. The following screen is displayed.
   
   

   

   
   
2. Find the alert in question, and select Silence.
   
   

   

   
   
3. Fill in the Creator and Comment details, and click Create. The alert should no longer show on the Monitoring Dashboard or be reported to any of the configured receivers.

Configuring the alerts

Note:

You can find uipathctl in the Automation Suite installation folder: .../UiPathAutomationSuite/UiPath_Installer/bin.

Before starting configuring the alerts, make sure to enable kubectl.

Adding a new email configuration

To add a new email configuration after an installation, run the following command:

./uipathctl config alerts add-email \
  --name test \
  --to "[e-mail]" \
  --from "[e-mail]" \
  --smtp server.mycompany.com \
  --username admin \
  --password somesecret \
  --require-tls \
  --ca-file <path_to_ca_file> \
  --cert-file <path_to_cert_file> \
  --key-file <path_to_key_file> \
  --send-resolved

Flag	Description	Example
name	The name of the email configuration	testconfig
to	The email address of the receiver	[e-mail]
from	The email address of the sender	[e-mail]
SMTP	SMTP server URL or IP address and port number	server.mycompany.com:567
username	Authentication username	admin
password	Authentication password	securepassword
require-tls	Boolean flag to denote that TLS is enabled at the SMTP server.	N/A
ca-file	File path containing the CA Certificate of the SMTP server. This is optional if the CA is private.	./ca-file.crt
cert-file	File path containing the certificate of the SMTP server. This is optional if the certificate is private.	./cert-file.crt
key-file	File path containing the private key of the certificate of the SMTP server. This is required if the certificate is private.	./key-file.crt
send-resolved	Boolean flag to send an email once the alert is resolved.	N/A

Removing an email configuration

To remove an email configuration, you must run the following command. Make sure to pass the name of the email configuration you want to remove.

./uipathctl config alerts remove-email --name test

Updating an email configuration

To update an email configuration, you must run the following command. Make sure to pass the name of the email configuration you want to update and the additional optional parameters you want to edit. These parameters are the same as the ones for adding a new email configuration. You can pass one or more flags at the same time.

./uipathctl config alerts update-email --name test [additional_flags]

Accessing Grafana dashboard

To access Grafana dashboards, you must retrieve your credentials and use them to log in:

• Username:

  kubectl -n cattle-monitoring-system get secrets/rancher-monitoring-grafana -o "jsonpath={.data.admin-user}" | base64 -d; echo

• Password:

  kubectl -n cattle-monitoring-system get secrets/rancher-monitoring-grafana -o "jsonpath={.data.admin-password}" | base64 -d; echo

Monitoring the Service Mesh

You can monitor the Istio Service Mesh via the following Grafana dashboards: Istio Mesh and Istio Workload.

Istio Mesh dashboard

This dashboard shows the overall request volume, as well as 400 and 500 error rates across the entire service mesh, for the selected time period. The data is displayed in the upper-right corner of the window. See the 4 charts across the top for this information.

It also shows the immediate Success Rate over the past minute for each individual service. Note that a Success Rate of NaN indicates the service is not currently serving traffic.

Istio Workload dashboard

This dashboard shows the traffic metrics over the time range selected in the upper-right corner of the window.

Use the selectors at the top of the dashboard to drill into specific workloads. Of particular interest is the uipath namespace.

The top section shows overall metrics, the Inbound Workloads section separates out traffic based on origin, and the Outbound Services section separates out traffic based on destination.

Monitoring Persistent Volumes

You can monitor persistent volumes via the Kubernetes / Persistent Volumes dashboard. You can keep track of the free and used space for each volume.

You can also check the status of each volume by clicking the PersistentVolumes item within the Storage menu of the Cluster Explorer.

Monitoring hardware utilization

To check the hardware utilization per node, you can use the Nodes dashboard. Data on the CPU, Memory, Disk, and Network is available.

You can monitor the hardware utilization for specific workloads using the Kubernetes / Compute Resources / Namespace (Workloads) dashboard. Select the uipath namespace to get the needed data.

Creating shareable visual snapshot of a Grafana chart

1. Click the downwards pointing arrow next to the chart title, and then select Share.
2. Click the Snapshot tab, and set the Snapshot name,Expire, and Timeout.
3. Click Publish to snapshot.raintank.io.

For more details, see the Grafana documentation on sharing dashboards.

Note: This snapshot is viewable on the public Internet by anyone with the link.

Creating custom persistent Grafana dashboards

For details on how to create custom persisten Grafana dashboards, see Rancher documentation.

Admin access to Grafana

Admin access to Grafana is not typically needed in Automation Suite clusters as dashboards are available for read access by default to anonymous users, and creating custom persistent dashboards must be created using the Kubernetes-native instructions linked above in this document.

Nonetheless, admin access to Grafana is possible with the instructions below.

The default username and password for Grafana admin access can be retrieved as follows:

kubectl get secret -n cattle-monitoring-system rancher-monitoring-grafana -o jsonpath='{.data.admin-user}' | base64 -d && echo
kubectl get secret -n cattle-monitoring-system rancher-monitoring-grafana -o jsonpath='{.data.admin-password}' | base64 -d && echo

Note that in High Availability Automation Suite clusters, there are multiple Grafana pods in order to enable uninterrupted read access in case of node failure, as well as a higher volume of read queries. This is incompatible with admin access because the pods do not share session state and logging in requires it. In order to work around this, the number of Grafana replicas must be temporarily scaled to 1 while admin access is desired. See below for instructions on how to scale the number of Grafana replicas:

# scale down
kubectl scale -n cattle-monitoring-system deployment/rancher-monitoring-grafana --replicas=1
# scale up
kubectl scale -n cattle-monitoring-system deployment/rancher-monitoring-grafana --replicas=2

Querying Prometheus

On the Monitoring Dashboard, click Prometheus Graph. A new window is displayed.

Documentation on the available metrics is here:

• https://github.com/prometheus/node\_exporter#enabled-by-default - Connect to preview
• https://github.com/kubernetes/kube-state-metrics/tree/master/docs#exposed-metrics - Connect to preview
• Querying Metrics from Prometheus (Istio)

Creating custom alerts

You can create custom alerts using a Prometheus query with a Boolean expression.

1. To do so, click Prometheus Rules in the Advanced menu of the Monitoring Dashboard.
2. Click Create in the upper-right corner of the window to create a new alert, and follow the Rancher documentation: PrometheusRules
3. When the alert fires, it should show on the Monitoring Dashboard. Additionally, it will be routed to any of the configured receivers.

Monitoring Kubernetes resource status

To see the status of pods, deployments, statefulsets, etc., you can use the Cluster Explorer UI. This is the same landing page as accessed after logging into the rancher-server endpoint. The homepage shows a summary, with drill downs into specific details for each resource type on the left. Note the namespace selector at the top of the page. This dashboard may also be replaced with the Lens tool.

Exporting Prometheus metrics to an external system

Prometheus uses the Prometheus remote write feature to collect and export Prometheus metrics to an external system.

Note: UiPath® does not support or maintain the remote write endpoint integrations. However, the endpoints are compatible with the Prometheus instance delivered within Automation Suite.

To configure remote_write on an Automation Suite cluster:

1. Connect to ArgoCD.
2. Click Applications.
3. Navigate to fabric-installer.
4. Open the APP DETAILS panel and disable self-heal.
5. Navigate to the rancher-monitoring application.
   
   

   

   
   
6. Open the APP DETAILS panel > MANIFEST tab.
   
   

   

   
   
7. Click EDIT and navigate to the values > prometheus > prometheusSpec section.
8. Add the desired remoteWrite configurations.
   Discover the available configurations for the remote write feature.

   
9. SAVE the new configuration. The rancher-monitoring application displays OutOfSync until the new configuration is applied.
   Note: Prometheus does not need to restart to apply the new remote write configurations.

   

   
   

   

   
   
10. Test the desired remote write integration. Return to step 8 to add a new configuration.