insights
latest
false
- Release Notes Cloud Insights
- Getting Started
- Access and Permissions
- Notifications
- Interacting with Insights
- Action Center Integration
- Automation Hub Integration
- Autopilot integration
- Document Understanding Integration
- License monitoring integrations
- Real-time Monitoring
- Real Time Data Export
- Troubleshooting
Sending data to Elasticsearch
Insights
Last updated Nov 28, 2024
Sending data to Elasticsearch
In this topic you can learn how to leverage the Insights real-time data export feature with AWS SQS to send data to ElasticSearch and use it to populate real-time monitoring data into Kibana dashboard.
Before you can start viewing real-time monitoring data in Kibana, the following hosts must be set up:
There is no out-of-the-box support for ElasticSearch consuming SNS messages. To do this, an SNS -> SQS -> Elastic pipeline needs to be configured. An SQS queue can be configured to be the output of an SNS topic. To do so, follow the instructions from the official AWS documentation.
To configure a new LogStash pipeline from the UI, login to Kibana user the
elastic
username and navigate to and use the following example for the pipeline
template:input {
sqs {
access_key_id => "<access_key_id>"
secret_access_key => "<secret_access_key>"
queue => "hztest_sns_sqs_output"
region => "us-west-2"
}
}
filter {
}
output {
elasticsearch {
index => "test_aws_sqs"
}
}
input {
sqs {
access_key_id => "<access_key_id>"
secret_access_key => "<secret_access_key>"
queue => "hztest_sns_sqs_output"
region => "us-west-2"
}
}
filter {
}
output {
elasticsearch {
index => "test_aws_sqs"
}
}
Note:
- The
regions
parameter is optional, but if missing the default region is set tous-east-1
, and the configuration will fail if the SQS is defined in a different region. - For secrets like ElastiscSearch password or AWS IAM credentials, consider using logstash-keystore.
- If a new index is created to be able to discover it do make sure to add the index to the indexes pattern via .
To test the configuration, run the following command on the LogStash, where
test_config
is a file with the pipeline definition above.
/usr/share/logstash/bin/logstash --config.test_and_exit -f ./test_config
/usr/share/logstash/bin/logstash --config.test_and_exit -f ./test_config