Exporting data from one elk cluster to another elk cluster

Elastic Stack Logstash
1

HI

i am trying to import data from one elk cluster to another elk cluster using logstash..but it gives error as below..(i am using logstash 2.4)..here is the logstash conf

input {
elasticsearch {
hosts => ["old_cluster_ip:9200"]
index => "lob*"
query => "*"
size => 500
scroll => "5m"
docinfo => true
}
}
output {
elasticsearch {
hosts => ["new_cluster_ip:9200", "new_cluster_ip:9200"]
index => "%{[@metadata][_index]}"
document_type => "%{[@metadata][_type]}"
document_id => "%{[@metadata][_id]}"
}
}

{:timestamp=>"2016-10-06T12:05:10.104000-0700", :message=>"A plugin had an unrecoverable error. Will restart this plugin.\n Plugin: <LogStash::Inputs::Elasticsearch hosts=>["old_cluster_ip:9200"], index=>"lob*", query=>"*", codec=><LogStash::Codecs::JSON charset=>"UTF-8">, scan=>true, size=>1000, scroll=>"1m", docinfo=>false, docinfo_target=>"@metadata", docinfo_fields=>["_index", "_type", "_id"], ssl=>false>\n Error: [400] {"error":{"root_cause":[{"type":"parse_exception","reason":"Failed to derive xcontent"}],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"init_scan","grouped":true,"failed_shards":[{"shard":0,"index":"lob20160630","node":"s4l2VJIsRvewpSYdnPZ8mQ","reason":{"type":"parse_exception","reason":"Failed to derive xcontent"}}]},"status":400}", :level=>:error}

any idea why this error is happening ?

2

Are both the same version? Check the ES logs, there's a 400 error there that should show something.

3

source elk cluster is @ 2.1.0 and destiination elk is @ 2.2.0..is that a problem ?

4

No that's fine.

5

seeing the below error in es logs at source side

[2016-10-07 10:41:54,777][DEBUG][action.search.type ] [elkstack1-cc] [lob20160913][4], node[s4l2VJIsRvewpSYdnPZ8mQ], [P], v[2], s[STARTED], a[id=BNwOlmwjRyesSsrFQV-cOg]: Failed to execute [org.elasticsearch.action.search.SearchRequest@705d939a] lastShard [true]
RemoteTransportException[[elkstack1-cc][:9300][indices:data/read/search[phase/scan]]]; nested: SearchParseException[failed to parse search source [na]]; nested: ElasticsearchParseException[Failed to derive xcontent];
Caused by: SearchParseException[failed to parse search source [na]]; nested: ElasticsearchParseException[Failed to derive xcontent];
at org.elasticsearch.search.SearchService.parseSource(SearchService.java:848)
at org.elasticsearch.search.SearchService.createContext(SearchService.java:651)
at org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:617)
at org.elasticsearch.search.SearchService.executeScan(SearchService.java:279)

any clue why its happening ?

6

Hello shankarbala,

Maybe you could use output { stdout { codec => rubydebug } } } instead of the current elasticsearch output so we can get a better idea of what is being read from the source elasticsearch.

Cheers.