How To Output to Amazon Elasticsearch Service

[mdanner (Martin Danner)]

Hello,

I'm setting up my first ELK stack on AWS. I installed Logstash 5.6.0 on an EC2 instance running Ubuntu 16.04.3 LTS. It was installed using the instructions for Debian/RPM. The input side of things is running fine. Now I want to output to Elasticsearch.

What's the best approach to output to the Amazon Elasticsearch service? Ideally I'd like some clear, easy to understand instructions for a newbie.

[Woodford]

Hey Martin:
Since you say you're a newbie, I'll assume that you're using AES with default settings, which means you need to 'sign' your requests to AES.

Amazon publishes an output filter plugin for Logstash that handles signing requests and makes it simple: https://github.com/awslabs/logstash-output-amazon_es

I haven't used it, so I can't comment on any current "gotchas", but it's actively maintained so you should be able to get help if you need it.

If you do run it successfully, you might want to post your step-by-step details here for the next newbie!

If you have setup AES to allow anonymous access (not recommended) you can configure Logstash outputs yourself. AWS provides simple examples here: http://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/es-kibana.html#es-managedomains-logstash

For example:

output{
elasticsearch {
hosts => "search-logs-demo0-cpxczkdpi4bkb4c44g3csyln5a.us-east-1.es.example.com"
ssl => true
flush_size => 250000
}
}

Good luck!

[mdanner (Martin Danner)]

Thanks for that, Woodward. I ran into trouble when trying to install the plug in you recommended. Simply put, the instructions on the README.md don't work.

Curious, as the Logstash configuration I implemented (Debian/RPM) appears to be the preferred option. Even more curious since the plugin is provided by Amazon, yet their installation instructions don't work and support for the plugin is nonexistent. You'd think they'd make it less difficult to use their ES service! Frustrating...

Correct guidance regarding the installation of the logstash-output-amazon-es plugin would be greatly appreciated.

[warkolm (Mark Walkom)]

You might have better luck using Elastic Cloud, then it's just the standard Elasticsearch output plugin and will work with the latest versions of the Elastic Stack.

[mdanner (Martin Danner)]

@warkolm

Elastic Cloud

Yes @warkolm, the Elastic Cloud is fantastic! I have some AWS credit, though, so I'm doing it the hard way.

I get the impression it might be easier just to stand up my own Elasticsearch server in the same VPC as the LogStash server.

Thoughts?

[mdanner (Martin Danner)]

I'm pleased to report that I installed the logstash-output-amazon-es plugin successfully. Here are the commands that worked:

cd /usr/share/logstash
sudo bin/logstash-plugin install logstash-output-amazon_es

It's worth noting that these commands are not documented in the README for this plugin.

https://github.com/awslabs/logstash-output-amazon_es/blob/master/README.md

I've made the developers aware of this problem - hopefully they'll update the documentation.