Reindexing with logstash in AWS ES service

Elastic Stack Elasticsearch
1

hi, i'm trying to use the elasticsearch input plugin to reindex some corrupted indexes per --https://gist.github.com/markwalkom/8a7201e3f6ea4354ae06

We are running AWS ES service and are running into issues using the elasticsearch input plugin. I'm talking with the elasticsearch cluster from the logstash machine that is currently piping data into it. So to confirm my current logstash instance can connect and indeed write to the elasticsearch clutser hosted by AWS as a service. So there is no connectivity issues at least to write new data. I can telnet to port 80 just fine from the logstash machine to the es domain/cluster (they run on port 80 as opposed to 9200.. or it probably behind some sort of load balancer or something).

When I start up the logstash instance with the new elasticsearch input config in there I receive this error:

A plugin had an unrecoverable error. Will restart this plugin.
  Plugin: <LogStash::Inputs::Elasticsearch hosts=>["*****************************.es.amazonaws.com"], port=>80, index=>"nginx-2015.11.17", size=>1000, scroll=>"5m", docinfo=>true, scan=>true, debug=>false, codec=><LogStash::Codecs::JSON charset=>"UTF-8">, query=>"{\"query\": { \"match_all\": {} } }", docinfo_target=>"@metadata", docinfo_fields=>["_index", "_type", "_id"], ssl=>false>
  Error: Connection refused - Connection refused {:level=>:error}

here is the output of telnet:

telnet *****************************.es.amazonaws.com 80
Trying ************...
Connected to *****************************.es.amazonaws.com.
Escape character is '^]'.

Is there something special happening with the elasticsearch input plugin? Anyone else come across this before too ?

2

My access policies are probably configured to allow all actions from this IP. I'm thinking it must be something to do with using this ES input plugin with AWS ES service. I have tried hitting AWS support for some answers 2 weeks ago and have been speaking with them since, but so far they don't have any answers.

3

Faced the same issue with you. But managed to worked around the issue. The best solution is basically to upgrade the to a bigger, better machine. Use the paramedic plugin to view the utilization. I give you my example.

How I approach it: Logstash Reindex Strategy