External Secrets troubleshooting

Solutions for External Secrets Operator issues in Automation Suite on EKS/AKS.

Overview

This section provides troubleshooting steps for resolving issues related to the External Secrets Operator and secretstore configuration in Automation Suite.

To identify and resolve synchronization or authentication issues between your cluster and the external secret provider, follow the steps described in this section.

Verify External Secrets pod status and logs

Begin by checking that the External Secrets Operator pods are running correctly.

If the pods are not in a Running state or show repeated restarts, review their logs for any errors related to authentication, permissions, or misconfiguration.

To check the status of the External Secrets pods and view their logs, run:

kubectl get pods -A | grep external-secrets
kubectl logs -f <pod-name> -n <namespace>

For example, to view logs from a specific pod, run:

kubectl logs -f external-secrets-7d8c9b5f9f-abc12 -n external-secrets

Inspect SecretStore and ExternalSecret objects

If the pods are running correctly, verify that the SecretStore and ExternalSecret resources are properly configured and synchronized.

These resources define how external secrets are retrieved and stored, so any misconfiguration can prevent successful synchronization.

• Check ExternalSecret resources Start by reviewing the ExternalSecret objects in your cluster. This helps confirm that the resources are correctly linked to their SecretStore and that the sync status is valid.
  • To display and inspect your ExternalSecret resources, run the following commands:

    kubectl get externalsecrets -A
    kubectl describe externalsecret <externalsecret-name> -n <namespace>
    

  • For example, to review a specific ExternalSecret in the uipath namespace:

    kubectl describe externalsecret azure-secret-store -n uipath
    

• Check SecretStore resources Inspect the associated SecretStore configuration. This resource defines the connection details for your secret provider (such as Azure Key Vault), and errors here can block secret synchronization.
  • To review the SecretStore definitions, use the following commands:

    kubectl get secretstores -A
    kubectl describe secretstore <secretstore-name> -n <namespace>
    

  • For example, to examine a SecretStore configuration in the uipath namespace:

    kubectl describe secretstore azure-secretstore -n uipath
    

• Check events related to External Secrets Review any Kubernetes events related to the External Secrets Operator. These events can reveal synchronization failures, permission issues, or provider misconfigurations.

  • To view related events, run the following command:

    kubectl get events -A | grep external-secrets
    

    When reviewing the event output, look for:

    • Error messages in the Status section
    • Conditions indicating whether Ready: True
    • Warnings or synchronization failures in the Events section

Verify Secret Store credentials

If you are using Azure Key Vault as your secret provider, ensure that the credentials and configuration are valid and up to date.

Incorrect credentials or missing permissions are common causes of synchronization failures.

• Authentication credentials
  • Confirm that the clientId, clientSecret, and tenantId values are correct.
  • Ensure the service principal has the required permissions for the Key Vault.
• Key Vault access policies
  • Verify Get and List permissions for secrets.
  • Ensure the Key Vault firewall settings allow access from your cluster.
• Workload identity configuration (if applicable)
  • Check that federated credentials are correctly configured.
  • Ensure the associated service account has the necessary annotations.
• Key Vault URI

  • Verify that the vaultUrl in your SecretStore configuration is correct.

    Example format:

    https://<key-vault-name>.vault.azure.net/
    

Verify Azure Key Vault service principal permissions

If your deployment uses a service principal, confirm that it has access to retrieve secrets from Azure Key Vault.

To list the secrets accessible to your service principal, run:

az keyvault secret list --vault-name <key-vault-name>

Verify HashiCorp Vault connectivity and authentication

If you are using HashiCorp Vault as your secret provider, verify the following:

• Test Vault server reachability Confirm that the Vault server URL in your SecretStore configuration is reachable from a node or jumphost that has network access to the cluster:

  curl -sk https://<vault-server>:8200/v1/sys/health
  

  A healthy Vault returns a JSON response with "initialized": true and "sealed": false. If the command times out or returns connection refused, check network policies and firewall rules between the cluster and the Vault server.

• Verify Vault seal status

  curl -sk https://<vault-server>:8200/v1/sys/seal-status
  

  If "sealed": true, the Vault server must be unsealed before ESO can retrieve secrets.

• Test Kubernetes Auth login If using auth_type: kubernetes, verify that a service account token can authenticate to Vault. First, extract the token from the ESO pod:

  ESO_POD=$(kubectl get pods -n uipath -l app.kubernetes.io/name=external-secrets -o jsonpath='{.items[0].metadata.name}')
  JWT=$(kubectl exec -n uipath $ESO_POD -- cat /var/run/secrets/kubernetes.io/serviceaccount/token)
  

  Then test the login:

  curl -sk --request POST \
    --data "{\"jwt\": \"$JWT\", \"role\": \"<vault-role>\"}" \
    https://<vault-server>:8200/v1/auth/<mount-path>/login
  

  A successful response contains a client_token. A permission denied error indicates the Vault role does not allow the service account, or the Kubernetes Auth method is misconfigured.

• Test AppRole Auth login If using auth_type: approle, verify the AppRole credentials:

  curl -sk --request POST \
    --data '{"role_id": "<role-id>", "secret_id": "<secret-id>"}' \
    https://<vault-server>:8200/v1/auth/<mount-path>/login
  

  If this fails, check that the AppRole secret-id has not expired (secret_id_ttl).

• Verify the AppRole credentials secret exists

  kubectl get secret <credentials-secret-name> -n uipath -o json | jq '.data | keys'
  

  The secret must contain both role-id and secret-id keys.

• Test reading a secret from Vault KV v2 Using a Vault token obtained from the auth steps above:

  curl -sk -H "X-Vault-Token: <vault-token>" \
    https://<vault-server>:8200/v1/<kv-path>/data/<secret-path>
  

  For KV v1, remove /data/ from the path. If the response is no value found at path, verify that hashicorp_kv.version matches the actual engine version.

• Verify KV engine version Ensure the hashicorp_kv.version in your configuration matches the actual KV engine version enabled in Vault (v1 or v2). A mismatch produces no value found at path errors because v1 and v2 use different API paths.

• Verify Kubernetes Auth configuration Ensure the Vault Kubernetes Auth method is enabled and configured with the correct kubernetes_host. Verify the Vault role allows the ESO service account (or custom service_account_name).

Verify AWS Secrets Manager connectivity and permissions

If you are using AWS Secrets Manager as your secret provider, verify the following:

• Verify the access key credentials secret exists

  kubectl get secret <credentials-secret-name> -n uipath -o json | jq '.data | keys'
  

  The secret must contain both access-key-id and secret-access-key keys.

• Test AWS Secrets Manager access Using the access key credentials, verify that the secret can be retrieved:

  aws secretsmanager get-secret-value \
    --secret-id <secret-name> --region <region>
  

  • AccessDeniedException - the IAM user does not have secretsmanager:GetSecretValue permission.
  • ResourceNotFoundException - the secret does not exist in the specified region. Verify the aws_kv.region value.

• Check region configuration Verify the aws_kv.region value matches the region where your secrets are stored. A region mismatch produces ResourceNotFoundException errors.

TLS and CA certificate issues

When the Vault or AWS endpoint uses HTTPS with a private CA certificate, verify the following:

• Check if the additional CA certificate secret exists

  kubectl get secret additional-ca-certificates -n uipath
  

• Verify the CA certificate content

  kubectl get secret additional-ca-certificates -n uipath -o jsonpath='{.data.ca\.crt}' | base64 -d | openssl x509 -text -noout
  

  Check that the certificate is not expired and that the issuer matches the CA used by your Vault or AWS endpoint.

• Test TLS handshake

  curl -v https://<vault-server>:8200/v1/sys/health
  

  If you see SSL certificate problem: unable to get local issuer certificate, the CA certificate is not being trusted. Verify that additional_ca_certs is configured in input.json. To test with a specific CA certificate file:

  curl -v --cacert /path/to/ca.crt https://<vault-server>:8200/v1/sys/health
  

Common configuration issues

Review the following common issues and corresponding error messages to help diagnose the problem.

Issue	Possible cause	Example / error output
Secret not found in cluster	Missing Kubernetes secret	Warning InvalidProviderConfig 3s (x4 over 8s) secret-store cannot get Kubernetes secret "azure-service-principal-secret" from namespace "uipath": secrets "azure-service-principal-secret" not found
Invalid credentials	Incorrect client ID, client secret, or tenant ID	error processing spec.data[0] (key: external-object-storage-account-key-10176524), err: azure.BearerAuthorizer#WithAuthorization: Failed to refresh the Token for request ... StatusCode=400
Missing Key Vault object	Key not present in Azure Key Vault	external-secrets error processing spec.data[0] (key: external-object-storage-account-key), err: Secret does not exist
HashiCorp Vault unreachable	Incorrect server URL or network policy blocking access	connection refused or context deadline exceeded
Vault auth failed	Kubernetes service account not bound to Vault role, or expired AppRole secret-id	permission denied
KV version mismatch	hashicorp_kv.version set to v2 but engine is v1 (or vice versa)	no value found at path
AWS credentials invalid	Incorrect access key ID or secret access key	InvalidClientTokenId or SignatureDoesNotMatch
AWS secret not found	Secret does not exist in the configured region, or region mismatch	ResourceNotFoundException

To further investigate these issues, review the controller logs or describe the related resources:

kubectl describe externalsecret <name> -n <namespace>
kubectl describe secretstore <name> -n <namespace>