Forwarding Kubernetes logs to Elasticsearch

Automation Suite supports forwarding Kubernetes infrastructure and application logs to external observability platforms.

This section describes how to forward logs to Elasticsearch, explains the available options, and outlines the recommended approach for most scenarios.

Hinweis:

This Section covers forwarding Kubernetes pod logs.

For forwarding Robot logs, refer to Saving robot logs to Elasticsearch.

Selecting a log forwarding option

Elastic provides multiple agents and operators for ingesting logs from Kubernetes. Depending on your Elastic Stack version, you can choose the most suitable mechanism:

• Filebeat (legacy approach)

  Filebeat was the traditional method for collecting Kubernetes logs and shipping them to Elasticsearch. It is suitable for older Elasticsearch versions (7.x and early 8.x).

  For more details, refer to the official Elastic documentation.

• Elastic Agent (unified agent)

  Elastic Agent collects all forms of data from anywhere with a single unified agent per host. It is recommended for Elastic Stack 8.x.

  For more details, refer to the official Elastic documentation.

• Elastic Distribution of OpenTelemetry (EDOT)

  EDOT (Elastic Distribution of OpenTelemetry) is recommended for new installations and for environments where you use Elasticsearch 8.16+.

EDOT is the recommended choice for the following reasons:

• Fully OpenTelemetry-compliant
• Developed and supported by Elastic
• Simple to operate (Helm charts and CRDs)
• Future-proof (Elastic has announced an OTel-first strategy)

To install EDOT in your Automation Suite cluster, follow the instructions provided in the official official Elastic documentation.

Customizing the EDOT configuration

You can modify the EDOT values file to match your environment and logging requirements.

Locate the corresponding sections in your EDOT values file and update them as needed. Apply only what is relevant to your setup.

Important: For full configuration options, refer to the official Elasticsearch and EDOT documentation.

The following examples show common customizations:

• To exclude noisy or high-volume logs, see the following configuration example:

  config:
    receivers:
      filelog:
        exclude:
            - /var/log/pods/opentelemetry-operator-system_opentelemetry-kube-stack*/*/*.log
            - /var/log/pods/*/*/istio-proxy/*.log
            - /var/log/pods/*/*/istio-init/*.log
            - /var/log/pods/*/*/aicenter-hit-count-update/*.log
            - /var/log/pods/*/*/istio-configure-executor/*.log
            - /var/log/pods/*/*/on-prem-tenant-license-update/*.log
            - /var/log/pods/*/*/curl/*.log
            - /var/log/pods/*/*/recovery/*.log
            - /var/log/pods/*/*/aicenter-oob-scheduler/*.log
            - /var/log/pods/*/*/cert-trustor/*.log
            - /var/log/pods/default_*/*.log
            - /var/log/pods/fleet-system_*/*.log
            - /var/log/pods/gatekeeper_*/*.log
  

• To configure TLS settings when you use self-signed certificates in a development setup, see the following configuration example:

  config:
    exporters:
      elasticsearch/otel:
        tls:
          insecure_skip_verify: true
  

  Use this only for non-production environments.
• To disable metrics and traces and use logs-only mode, see the following configuration example:

  config:
    service:
      pipelines:
        metrics: null
        traces: null
  

Deploying EDOT adds components (such as collectors, processors, and exporters) that run across the cluster. As a result, overall CPU and memory usage increases. Ensure that your cluster has enough available resources to support these components. You can adjust the EDOT resource requests and limits in the values file to meet your cluster’s requirements.