# Logstash 5.5.1 input kafka 0.9 creates a massive duplication of events **URL:** \ **Category:** Logstash\ **Created:** [September 6, 2017, 11:20am UTC](https://discuss.elastic.co/t/logstash-5-5-1-input-kafka-0-9-creates-a-massive-duplication-of-events/99568 "2017-09-06T11:20:29Z")\ **Posts on this page:** 2\ **Page:** 1
**Author:** ![JMNDev](https://avatars.discourse-cdn.com/v4/letter/j/3e96dc/32.png) [@JMNDev](https://discuss.elastic.co/u/JMNDev)\ **Post date:** [September 6, 2017, 11:20am UTC](https://discuss.elastic.co/t/logstash-5-5-1-input-kafka-0-9-creates-a-massive-duplication-of-events/99568/1 "2017-09-06T11:20:29Z")
I have the following case. 1 filebeat sending events to kafka then 1 logstash reading from kafka and sending to elasticsearch What is happenning is a massive duplication of events in some random periodicity in Elasticsearch. I have validated and the files are being correctly put into kafka by the filebeat, the offset in kafka is also correct but sometimes the logstash duplicates events. I tried to use console consumer i only get 1 of each event. I havel also tried to adjust heartbeat\_timeout\_ms and i got less duplication but i would like to cut all the duplication The versions I'm using are: filebeat 5.0.0 kafka 0.9 logstash 5.5.1 with logstash-input-kafka (4.2.0) the logstash input configuration I'm using is the following: > input { > kafka { > topics =\> ["xxxxxxx"] > group\_id =\> "logstash-xxxxxx" > bootstrap\_servers =\> "xxxxxxx:6667,xxxxxxxxx:6667" > consumer\_threads =\> 4 > heartbeat\_interval\_ms =\> "500" > auto\_commit\_interval\_ms =\> "100" > codec =\> "json" > } > } Can someone help me , Thanks ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [October 4, 2017, 11:20am UTC](https://discuss.elastic.co/t/logstash-5-5-1-input-kafka-0-9-creates-a-massive-duplication-of-events/99568/2 "2017-10-04T11:20:50Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.