Copying data from 1 es instance to another using logstash


(Ramdev Wudali) #1

Hi:
I have been trying to use logstash to copy data from one ES instance to another ES instance. However I keep running into the following problem :

A plugin had an unrecoverable error. Will restart this plugin.
  Plugin: <LogStash::Inputs::Elasticsearch hosts=>["porto.int.thomsonreuters.com", "gnosis.int.thomsonreuters.com", "olimp.int.thomsonreuters.com"], index=>".kibana-tribe", scroll=>"5m", docinfo=>true, codec=><LogStash::Codecs::JSON charset=>"UTF-8">, query=>"{\"query\": { \"match_all\": {} } }", scan=>true, size=>1000, docinfo_target=>"@metadata", docinfo_fields=>["_index", "_type", "_id"], ssl=>false>
  Error: Connection refused - Connection refused {:level=>:error}
A plugin had an unrecoverable error. Will restart this plugin.
  Plugin: <LogStash::Inputs::Elasticsearch hosts=>["porto.int.thomsonreuters.com", "gnosis.int.thomsonreuters.com", "olimp.int.thomsonreuters.com"], index=>".kibana-tribe", scroll=>"5m", docinfo=>true, codec=><LogStash::Codecs::JSON charset=>"UTF-8">, query=>"{\"query\": { \"match_all\": {} } }", scan=>true, size=>1000, docinfo_target=>"@metadata", docinfo_fields=>["_index", "_type", "_id"], ssl=>false>
  Error: Connection refused - Connection re  

At this point logstash retries and starts at the beginning of the ES index and it does that every time the above error occurs. So what ends up happening is the same data keeps getting indexed over and over again without a lot of progress...

I am using ES 1.5.2 with logstash 1.5.1 (I tried it with 1.5.4 and 2.0.0 as well) the same problem.

Any help with this is appreciated.

Thanks much

Ramdev


(system) #2