Insights
latest
false
Banner background image
Insights
Last updated Feb 23, 2024

Sending Data to Elasticsearch

Overview

In this topic you can learn how to leverage the Insights real-time data export feature to send data to ElasticSearch and use it to populate real-time monitoring data into Kibana dashboard.

Prerequisites

Before you can start viewing real-time monitoring data in Kibana, the following hosts must be set up:

Setting up an ElasticSearch Host

You can choose to host ElasticSearch yourself or to use the cloud option.

For more information, check the official Elastic website.

Setting up a FileBeat Host

Overview

FileBeat is a tool used to gather data from a source (EventHub in this case) and send it into ElasticSearch.

Installation

You need to host your own FileBeat server.

To install FileBeat, first provision your hosts and follow the instructions from the official Elastic website for your operating system.

Configuration

To configure FileBeat, use the following instructions:

  1. Open the filebeat.yml file.
  2. Set up a connection to ElasticSearch:

    cloud.id: "cloud:YOUR_CLOUD_ID"
    cloud.auth: "USERNAME:YOUR_PASSWORD"cloud.id: "cloud:YOUR_CLOUD_ID"
    cloud.auth: "USERNAME:YOUR_PASSWORD"

    You can get your cloud ID from the Deployments tab in ElasticSearch.



  3. Go to the filebeat.inputs section in the filebeat.yml file.
  4. Add your EventHub connection settings in the filebeat.inputs section.
    <ul>
    <li>type: azure-eventhub
      enabled: true
      eventhub: <YOUR EVENTHUB>
      consumer_group: "filebeat"
      connection_string: <YOUR CONNECTION STRING>
      storage_account_key: <YOUR STORAGE ACCOUNT KEY>
      storage_account_container: ""
      resource_manager_endpoint: """</li>
    </ul><ul>
    <li>type: azure-eventhub
      enabled: true
      eventhub: <YOUR EVENTHUB>
      consumer_group: "filebeat"
      connection_string: <YOUR CONNECTION STRING>
      storage_account_key: <YOUR STORAGE ACCOUNT KEY>
      storage_account_container: ""
      resource_manager_endpoint: """</li>
    </ul>
  5. Go to the processors section in the filebeat.yml file.
  6. Add the following lines in the processors section:
    <ul>
    <li>decode_json_fields:
          fields: ["message"]
          process_array: false
          max_depth: 2
          target: ""
          overwrite_keys: false
          add_error_key: true</li>
    </ul><ul>
    <li>decode_json_fields:
          fields: ["message"]
          process_array: false
          max_depth: 2
          target: ""
          overwrite_keys: false
          add_error_key: true</li>
    </ul>

Starting up the FileBeat service

After you successfully installed and configured FileBeat, follow the procedure from the official Elastic website to start the service.

View Data in Kibana

To view your data in Kibana, use the following steps.

  1. Open ElasticSearch.
  2. Expand the Analytics category from the dashboard on the left.
  3. Select Discover.



  4. Set the data view to filebeat-*.



You can now view your data in Kibana.



  • Overview
  • Prerequisites
  • Setting up an ElasticSearch Host
  • Setting up a FileBeat Host
  • View Data in Kibana

Was this page helpful?

Get The Help You Need
Learning RPA - Automation Courses
UiPath Community Forum
Uipath Logo White
Trust and Security
© 2005-2024 UiPath. All rights reserved.