# Logstash Kafka input causes duplicates

**URL:** <https://discuss.elastic.co/t/logstash-kafka-input-causes-duplicates/96885>\
**Category:** Logstash\
**Created:** [August 13, 2017, 6:56pm UTC](https://discuss.elastic.co/t/logstash-kafka-input-causes-duplicates/96885 "2017-08-13T18:56:37Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![guyisra](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/guyisra/32/11912_2.png) [@guyisra](https://discuss.elastic.co/u/guyisra)\
**Post date:** [August 13, 2017, 6:56pm UTC](https://discuss.elastic.co/t/logstash-kafka-input-causes-duplicates/96885/1 "2017-08-13T18:56:37Z")

</div>

I am using Logstash 5.5.1 with kafka input 5.1.8.  
When consuming the topic, logstash forwards many duplicates to the output, while the event it self was consumed only once.

Is this a bug or a configuration issue?

This causing me to believe logstash is not reliable as a kafka consumer, since using other libraries that consume from kafka do not exhibit this kind of issues...

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [August 13, 2017, 7:23pm UTC](https://discuss.elastic.co/t/logstash-kafka-input-causes-duplicates/96885/2 "2017-08-13T19:23:59Z")

</div>

Seeing the smallest possible configuration that reproduces the issue would probably be helpful to those who know Kafka and the kafka input plugin.

---

<div class="post-metadata">

**Author:** ![guyisra](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/guyisra/32/11912_2.png) [@guyisra](https://discuss.elastic.co/u/guyisra)\
**Post date:** [August 15, 2017, 10:56am UTC](https://discuss.elastic.co/t/logstash-kafka-input-causes-duplicates/96885/3 "2017-08-15T10:56:21Z")

</div>

This is the logstash.conf. Workers and batch size are default (to 5.5.1). The topics that are being consumed produce 100's of thousands events a day

```
input {
  kafka {
    bootstrap_servers => "${KAFKA_BROKER_ADDRESS}"
    topics_pattern => "${KAFKA_CONSUMED_TOPICS_PATTERN}"
  }
}

filter {
  json {
    source => "message"
  }
}

output {
  http {
    url => "${URL}"
    http_method => 'post'
    socket_timeout => 180
    pool_max_per_route => 50
    automatic_retries => 5
  }
}
```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [September 12, 2017, 10:56am UTC](https://discuss.elastic.co/t/logstash-kafka-input-causes-duplicates/96885/4 "2017-09-12T10:56:42Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.