# Kafka input plugin creates anywhere from 2-5x more documents than in topic

**URL:** <https://discuss.elastic.co/t/kafka-input-plugin-creates-anywhere-from-2-5x-more-documents-than-in-topic/83449>\
**Category:** Logstash\
**Created:** [April 24, 2017, 6:18pm UTC](https://discuss.elastic.co/t/kafka-input-plugin-creates-anywhere-from-2-5x-more-documents-than-in-topic/83449 "2017-04-24T18:18:39Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![xrl](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/xrl/32/3556_2.png) [@xrl](https://discuss.elastic.co/u/xrl)\
**Post date:** [April 24, 2017, 6:18pm UTC](https://discuss.elastic.co/t/kafka-input-plugin-creates-anywhere-from-2-5x-more-documents-than-in-topic/83449/1 "2017-04-24T18:18:39Z")

</div>

I am trying to run a small pool of logstash with at-most-once delivery semantics on a kafka 0.10.2.1 cluster. The kafka input client is running with the new style of kafka consumer group, managed as metadata in the brokers, and it does not use Zookeeper for offset tracking. I have a brief commit interval to keep the offsets moving between consumers.

```auto
input {
  kafka {
    bootstrap_servers => "${KAFKA_BOOTSTRAP_SERVERS}"
    topics => ["proofpoint", "fortigate", "windows"]
    group_id => "logstash"
    auto_offset_reset => "latest"
    auto_commit_interval_ms => "50"

    codec => "json"
    consumer_threads => 1
    partition_assignment_strategy => "org.apache.kafka.clients.consumer.RoundRobinAssignor"
  }
}

```

My elasticsearch output config:

```auto
output {
    elasticsearch {
      hosts => ["${ELASTICSEARCH_CLIENT_HOST}"]
      index => "@%{vtype}-nad-v2-%{date}"
      flush_size => 1000
      idle_flush_time => 10
      sniffing => true
      sniffing_delay => 30
    }
}

```

Now I see in Elasticsearch that I have 5x the data from what is in kafka. What am I doing wrong?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 22, 2017, 6:25pm UTC](https://discuss.elastic.co/t/kafka-input-plugin-creates-anywhere-from-2-5x-more-documents-than-in-topic/83449/2 "2017-05-22T18:25:43Z")

</div>

Closed on May 22, 2017, 6:25pm UTC

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.