Sending data to Elasticsearch

In this topic you can learn how to leverage the Insights real-time data export feature with AWS SQS to send data to ElasticSearch and use it to populate real-time monitoring data into Kibana dashboard.

Before you can start viewing real-time monitoring data in Kibana, the following hosts must be set up:

• Set up AWS SQS

Configure ElasticSearch for AWS SQS

There is no out-of-the-box support for ElasticSearch consuming SNS messages. To do this, an SNS -> SQS -> Elastic pipeline needs to be configured. An SQS queue can be configured to be the output of an SNS topic. To do so, follow the instructions from the official AWS documentation.

To configure a new LogStash pipeline from the UI, login to Kibana user the elastic username and navigate to Manage > LogStash Pipeline > Create Pipeline and use the following example for the pipeline template:

input {
    sqs {
        access_key_id => "AKIARCQHHGMYMN57UFFT"
        secret_access_key => "0sGGKEJUg4/mayyCw38hvZNY3BzeSuK1Fk6DrIwv"
        queue => "hztest_sns_sqs_output"
        region => "us-west-2"
      }
}
filter {
}
output {
    elasticsearch {
        index => "test_aws_sqs"
      }
}

Note:

• The regions parameter is optional, but if missing the default region is set to us-east-1, and the configuration will fail if the SQS is defined in a different region.
• For secrets like ElastiscSearch password or AWS IAM credentials, consider using logstash-keystore.
• If a new index is created to be able to discover it do make sure to add the index to the indexes pattern via Stack Management > Index Patters > create Index pattern .

To test the configuration, run the following command on the LogStash, where test_config is a file with the pipeline definition above.

/usr/share/logstash/bin/logstash --config.test_and_exit -f ./test_config