Logstash elastic search on ec2 (cloud plugin)

Elastic Stack Logstash
1

Hi:

I was just wondering if logstash can auto discover elastic search nodes using ec2 discovery on aws. For elastic search we need to use elastic search cloud aws plugin (https://github.com/elastic/elasticsearch-cloud-aws). But I could not find any documentation on how to enable logstash's elastic search output plugin with the ec2 discovery feature.

If you have any pointers, please let me know.

Thanks

Mans

2

One way I can suggest (but not really an answer to your original question) is use ELB in front of ES. If you autoscale + ELB in front of ES stack, you will get load balance. The benefit of ELB is you can attach EC2 to it. Another way is have your cloudformation stack to update your DNS (A record in this case) in Route 53, so that logstash is only given "myelasticsearchcluster.xyz.com" and it can do cheap round-robin if you don't want to pay the extra cost of ELB.

3

Hello. I accidentally discovered this plugin https://github.com/logstash-plugins/logstash-output-elasticsearch-ec2 because it is not listed in the docs https://www.elastic.co/guide/en/logstash/current/output-plugins.html or anywhere else except of github. As well there's no documentation for it too. Plugin's options turned out to be here https://github.com/logstash-plugins/logstash-output-elasticsearch-ec2/blob/f005454d28ce31dc152724e526a6f79033bd9331/lib/logstash/outputs/elasticsearch/ec2.rb and a configuration like below worked for me meaning that logstash host discovered the cluster and successfully joined:

output {
    elasticsearch {
      cluster => "dev-elk-cluster-eu-west-1.aws"
      index => "dev-elk-index-eu-west-1.aws-%{+YYYY.MM.dd}"
      discovery => "ec2"
      aws_region => "eu-west-1"
      aws_protocol => "http"
      ec2_protocol => "http"
    }
}

Plugin developers must definitely include at least initial information on their usage apart from the default readme.md.