A plugin had an unrecoverable error. Will restart this plugin(logstash input elasicsearch)


(showmeall) #1

we use logstash 6.5.0 for ES reindex , reindex ES 2.4 TO ES 6.2; but always get a problem, the detail info as fllow:
[ERROR] 2018-11-18 12:15:21.938 [[main]<elasticsearch] pipeline - A plugin had an unrecoverable error. Will restart this plugin.
Pipeline_id:main
Plugin: <LogStash::Inputs::Elasticsearch size=>10000, hosts=>["10.10.10.100:9200"], query=>"{"query": {"filtered": { "query": {"term": { "_type": "domain" } },"filter" : {"range": {"lastupdatetime": {"gte": "2015-09-01 00:00:00","lt": "2015-10-01 00:00:00"}}}}}}", scroll=>"5m", index=>"test", docinfo=>true, id=>"e1dc09e8947eeb59b51f5fabdea40608d484466d84f0d65bcd133b730cd3c18f", enable_metric=>true, codec=><LogStash::Codecs::JSON id=>"json_226934a8-12cc-4a7c-a2ca-5604407e8e07", enable_metric=>true, charset=>"UTF-8">, docinfo_target=>"@metadata", docinfo_fields=>["_index", "_type", "_id"], ssl=>false>
Error: end of file reached
Exception: Faraday::ConnectionFailed
Stack:

due to this issue , can not finished reindex , it always restart and index the old data again and again,
it is bug for this plugin , or it is ES problem? please help us .
by the way , the ES have 10 billion docs and 10 TB store data。


(showmeall) #2

logstash config as follow
input {
elasticsearch {
hosts => [ "10.10.10.100:9200" ]
index => "test2"
query => '{"query": {"filtered": { "query": {"term": { "_type": "test2" } },"filter" : {"range": {"lastupdatetime": {"gte": "2015-09-01 00:00:00","lt": "2015-10-01 00:00:00"}}}}}}'
size => 10000
scroll => "5m"
docinfo => true
}
}
filter {
mutate {
remove_field => ["@timestamp", "@version"] #过滤掉logstash 自己加上的字段
}
}
output {
elasticsearch {
hosts => [ "10.10.10.200:9200" ]
index => "test2"
document_type => "%{[@metadata][_type]}"
document_id => "%{[@metadata][_id]}"
}


(system) #3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.