# Logstash doesn't stop outputting duplicates

**URL:** <https://discuss.elastic.co/t/logstash-doesnt-stop-outputting-duplicates/88890>\
**Category:** Logstash\
**Created:** [June 9, 2017, 8:30pm UTC](https://discuss.elastic.co/t/logstash-doesnt-stop-outputting-duplicates/88890 "2017-06-09T20:30:56Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![jacoblemperle](https://avatars.discourse-cdn.com/v4/letter/j/96bed5/32.png) [@jacoblemperle](https://discuss.elastic.co/u/jacoblemperle)\
**Post date:** [June 9, 2017, 8:30pm UTC](https://discuss.elastic.co/t/logstash-doesnt-stop-outputting-duplicates/88890/1 "2017-06-09T20:30:56Z")

</div>

I am receiving input from elasticsearch and outputting to RabbitMQ, there are about 1300 items that should be output to Rabbit but logstash keeps running it i had to cancel at 50000 items posted. I was hoping someone knew what could be causing this. as well I am getting this error

[2017-06-09T14:12:31,771][ERROR][logstash.pipeline] A plugin had an unrecoverable error. Will restart this plugin.  
Plugin: \<LogStash::Inputs::Elasticsearch index=\>”index”, hosts=\>["localhost"], query=\>"{ "query": { "bool": { "must": [{ "multi\_match" : { "query": \”1234\”, "type": "phrase", "fields": [ \”topic\”, \”other\”] } } ], "should": [{ "range": { \”date\”: { "gte": "2012-01-01", "lt": "2017-06-01" } } }, { "range": { \”time\”: { "gte": 1451610000000, "lte": 1496275199000 } } }], "minimum\_should\_match" : 1 } } }", scroll=\>"1m", docinfo=\>true, id=\>"ba1340421fca139b8466b4e5f0b4a957ae4c89a8-1", enable\_metric=\>true, codec=\>\<LogStash::Codecs::JSON id=\>"json\_801cdbd8-ffe4-46b3-a695-8557e643118e", enable\_metric=\>true, charset=\>"UTF-8"\>, size=\>1000, docinfo\_target=\>"@metadata", docinfo\_fields=\>["\_index", "\_type", "\_id"], ssl=\>false\>  
Error: [400] {"error":"ElasticsearchIllegalArgumentException[Failed to decode scrollId]; nested: IOException[Bad Base64 input character decimal 123 in array position 0]; ","status":400}  
[2017-06-09T14:12:35,356][ERROR][logstash.pipeline] A plugin had an unrecoverable error. Will restart this plugin.

input {  
elasticsearch {  
index =\> “index1”  
hosts =\> ["localhost"]  
query =\> '{ "query": { "bool": { "must": [{ "multi\_match" : { "query": "54815", "type": "phrase", "fields": [ “second thing”, “firstling”] } } ], "should": [{ "range": { “date”: { "gte": "2012-01-01", "lt": "2017-06-01" } } }, { "range": { “time”: { "gte": 1451610000000, "lte": 1496275199000 } } }], "minimum\_should\_match" : 1 } } }'  
scroll =\> "1m"  
docinfo =\> true  
}  
}  
filter {  
ruby {  
code =\> '  
event.set(“[setitem]”, event.get(“[item]”))  
event.set(“[seconditem]”, event.get(“[2item]”))  
}  
prune {  
whitelist\_names =\> [  
“Setter”, “second item”  
]  
}  
}  
output {  
rabbitmq {

}  
}

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 7, 2017, 8:30pm UTC](https://discuss.elastic.co/t/logstash-doesnt-stop-outputting-duplicates/88890/2 "2017-07-07T20:30:57Z")

</div>

Closed on July 7, 2017, 8:30pm UTC

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.