I am receiving input from elasticsearch and outputting to RabbitMQ, there are about 1300 items that should be output to Rabbit but logstash keeps running it i had to cancel at 50000 items posted. I was hoping someone knew what could be causing this. as well I am getting this error
[2017-06-09T14:12:31,771][ERROR][logstash.pipeline ] A plugin had an unrecoverable error. Will restart this plugin.
Plugin: <LogStash::Inputs::Elasticsearch index=>”index”, hosts=>["localhost"], query=>"{ "query": { "bool": { "must": [ { "multi_match" : { "query": \”1234\”, "type": "phrase", "fields": [ \”topic\”, \”other\” ] } } ], "should": [ { "range": { \”date\”: { "gte": "2012-01-01", "lt": "2017-06-01" } } }, { "range": { \”time\”: { "gte": 1451610000000, "lte": 1496275199000 } } } ], "minimum_should_match" : 1 } } }", scroll=>"1m", docinfo=>true, id=>"ba1340421fca139b8466b4e5f0b4a957ae4c89a8-1", enable_metric=>true, codec=><LogStash::Codecs::JSON id=>"json_801cdbd8-ffe4-46b3-a695-8557e643118e", enable_metric=>true, charset=>"UTF-8">, size=>1000, docinfo_target=>"@metadata", docinfo_fields=>["_index", "_type", "_id"], ssl=>false>
Error: [400] {"error":"ElasticsearchIllegalArgumentException[Failed to decode scrollId]; nested: IOException[Bad Base64 input character decimal 123 in array position 0]; ","status":400}
[2017-06-09T14:12:35,356][ERROR][logstash.pipeline ] A plugin had an unrecoverable error. Will restart this plugin.
input {
elasticsearch {
index => “index1”
hosts => ["localhost"]
query => '{ "query": { "bool": { "must": [ { "multi_match" : { "query": "54815", "type": "phrase", "fields": [ “second thing”, “firstling” ] } } ], "should": [ { "range": { “date”: { "gte": "2012-01-01", "lt": "2017-06-01" } } }, { "range": { “time”: { "gte": 1451610000000, "lte": 1496275199000 } } } ], "minimum_should_match" : 1 } } }'
scroll => "1m"
docinfo => true
}
}
filter {
ruby {
code => '
event.set(“[setitem]”, event.get(“[item]”))
event.set(“[seconditem]”, event.get(“[2item]”))
}
prune {
whitelist_names => [
“Setter”, “second item”
]
}
}
output {
rabbitmq {
}
}