Temporal as a Service troubleshooting

Solutions for Temporal as a Service issues in Automation Suite on EKS/AKS.

Maestro workflows are stuck or not starting

Symptom: Maestro process instances remain in a pending or running state without progressing.

Cause: TaaS pods are unavailable or unhealthy, preventing workflow tasks from being dispatched or picked up by workers.

Resolution

1. Check the status of all TaaS pods:

   kubectl get pods -n uipath | grep taas
   

2. Identify any pods that are not in Running state and review their logs:

   kubectl logs -n uipath <pod-name> --tail=100
   

3. Look for warn and error level entries. Common causes include database connectivity failures and resource exhaustion.
4. If the taas-temporal-history pod is restarting, check memory usage. The history service is memory-intensive and its autoscaler is configured on memory utilization.

TaaS pods are crash-looping

Symptom: One or more taas-temporal-* pods are in CrashLoopBackOff state.

Cause: The most common cause is a failure to connect to the PostgreSQL database on startup.

Resolution

1. Review the logs of the affected pod:

   kubectl logs -n uipath <pod-name> --previous
   

2. Check for connection errors referencing the persistence or visibility store.
3. Verify the PostgreSQL server is reachable from the cluster and that credentials in the taas-temporal-app-secrets secret are correct.
4. Confirm that the PostgreSQL max_connections value meets the minimum requirement for your TaaS profile. The default PostgreSQL value of 100 is insufficient. For the required values per profile, refer to SQL database.

Workflow execution is slow

Symptom: Maestro workflows start but tasks take longer than expected to be picked up or completed.

Cause: TaaS is under-resourced, the PostgreSQL connection pool is exhausted, or the number of worker pollers is insufficient for the current load.

Resolution

1. Check Prometheus metrics for temporal_activity_schedule_to_start_latency_bucket and temporal_workflow_task_schedule_to_start_latency_bucket. High values indicate tasks are waiting in the queue longer than expected.
2. Check that the PostgreSQL instance has sufficient available connections. Connection pool exhaustion causes persistence timeouts under load.
3. Review pod resource usage for taas-temporal-history and taas-temporal-matching. If pods are CPU- or memory-constrained, consider adjusting resource limits or scaling the deployment.

Accessing the Temporal UI for diagnostics

The Temporal web UI (taas-temporal-web) is not exposed outside the cluster by default. To access it for diagnostic purposes, port-forward to the pod:

kubectl port-forward -n uipath deployment/taas-temporal-web 8008:8008

Open http://localhost:8008 in a browser. No login is required when accessing via port-forward.

The Temporal UI lets you inspect individual workflow executions, view execution history, and identify failed or timed-out activities.

Lock contention errors in TaaS logs

Symptom: TaaS logs contain errors referencing lock contention or conflicting database writes.

Cause: Two Kubernetes clusters are connected to the same PostgreSQL persistence store simultaneously. Only one cluster can actively run TaaS at a time.

Resolution

In an active/passive deployment, scale all TaaS deployments on the passive cluster to zero replicas. For details on active/passive requirements for TaaS, refer to Overview.

TaaS init job fails in a proxy environment

Description

When Maestro or Temporal as a Service (TaaS) is enabled in a proxy environment, the TaaS init job fails with the following error:

Unable to get namespace default info ... context deadline exceeded

The taas-temporal-frontend, taas-temporal-history, and taas-temporal-internal-frontend pod logs contain the following error:

failed to do connect handshake ... HTTP/1.1 503

TaaS services connect to each other by pod IP address. If the no_proxy list does not cover a pod IP address, the connection goes to the proxy, which cannot reach pod IP addresses and rejects the request. This happens when pod IP addresses fall outside the ranges in the no_proxy list, for example with EKS custom networking or AKS with Azure CNI Overlay.

Solution

To address the issue, take the following steps:

1. Add the pod CIDR ranges and the node subnet CIDR ranges to proxy.no_proxy in input.json. For details, refer to Proxy.

2. Apply the configuration:

   uipathctl manifest apply input.json --versions versions.json