Setting up Splunk

Note:

• This section covers exporting POD logs. For exporting robot logs, see Ochestrator - About Logs.

• Splunk is an external tool, and UiPath does not have an opinion on how you should configure your Splunk setting. For more details about HTTP Event Collector, see Splunk official documentation.

The Splunk-Fluentd stack is a centralized logging solution that allows you to search, analyze, and visualize log data. Fluentd collects and sends the logs to Splunk. Splunk retrieves the logs and lets you visualize and analyze the data.

Creating a Secret With a Token

To create a secret in the cluster with Splunk's HTTP Event Collector token, run the following command:

kubectl -n cattle-logging-system create secret generic splunk-hec-token --from-literal=splunk_hec_token=<splunk_hec_token>

ClusterOutput to Splunk

Run the following command to ClusterOutput to Splunk. Replace the following attributes with the ones used in your Splunk configuration:

kubectl -n cattle-logging-system apply -f - <<"EOF"
apiVersion: logging.banzaicloud.io/v1beta1
kind: ClusterOutput
metadata:
  name: splunk-output
spec:
  splunkHec:
    buffer:
      tags: '[]'
      timekey: <splunk_hec_timekey>
      timekey_use_utc: true
      timekey_wait: 10s
      type: file
    hec_host: <splunk_hec_host>
    hec_port: <splunk_hec_port>
    hec_token:
      valueFrom:
        secretKeyRef:
          key: <secret_key>
          name: splunk-hec-token
    index: <splunk_hec_index>
    insecure_ssl: true
    protocol: <splunk_hec_protocol>
    source: <splunk_hec_source>
    sourcetype: <splunk_hec_source_type>
EOF

Attribute	Description
splunk_hec_host	The network host of your Splunk instance.
splunk_hec_port	The Splunk port for client communication.
secret_key	The secret key with the Splunk token.
splunk_hec_timekey value in splunkHec.buffer	The output frequency, i.e., how often you want to push logs.
protocol	The URL protocol. Valid values are http and https.
splunk_hec_index	Identifier for the Splunk index to be used for indexing events.
splunk_hec_source	The source field for events.
splunk_hec_source_type	The source type field for events.

ClusterFlow in Fluentd

Run the following command to ClusterFlow in Fluentd:

kubectl -n cattle-logging-system apply -f - <<"EOF"
apiVersion: logging.banzaicloud.io/v1beta1
kind: ClusterFlow
metadata:
  name: splunk-flow
  namespace: cattle-logging-system
spec:
  filters:
  - tag_normaliser:
      format: ${namespace_name}/${pod_name}.${container_name}
  globalOutputRefs:
  - splunk-output
  match:
  - select:
      container_names:
      - istio-proxy
      namespaces:
      - istio-system
  - exclude:
      container_names:
      - istio-proxy
      - istio-init
      - aicenter-hit-count-update
      - istio-configure-executor
      - on-prem-tenant-license-update
      - curl
      - recovery
      - aicenter-oob-scheduler
      - cert-trustor
  - exclude:
      namespaces:
      - fleet-system
      - cattle-gatekeeper-system
      - default
  - exclude:
      labels:
        app: csi-snapshotter
  - exclude:
      labels:
        longhorn.io/job-task: backup
  - exclude:
      labels:
        app: csi-resizer
  - select: {}
EOF

Note: Logs from the cluster are collected and logged to the ClusterOutput.

Configuring Splunk

To configure Splunk, take the following steps:

1. Click Settings in the top navigation bar, and then select Indexes.
   
   

   

   
   
2. Click New Index and then Create an index.
   
   

   

   
   

   
   

   

   
   
3. Click Settings in the top navigation bar, and then select Data Inputs.
   
   

   

   
   
4. Click HTTP Event Collector.
   
   

   

   
   
5. Click Global Settings to enable the new token creation.
   
   

   

   
   
6. Enable and save the Global Settings.
   
   

   

   
   
7. Click New Token to create the token.
   
   

   

   
   
8. Enter a name for the HTTP Event Collector and click Next.
   
   

   

   
   
9. Click New and enter Source Type details.
   
   

   

   
   
10. Scroll down and select Index from the available list of indexes, and click Next in the top navigation bar.
    
    

    

    
    
11. Verify the data and click Submit.
    
    

    

    
    
12. Once created, fetch the details of Token ID,Index,Source,Source Type. You will need these when setting up ClusterOutput.
    
    

    

    
    

Searching in Splunk

1. Click Search & Reporting.
   
   

   

   
   
2. Search based on Source,Index, and SourceType.