# Each time logstash instance dies, the number of data stored in elasticsearch increases

**URL:** <https://discuss.elastic.co/t/each-time-logstash-instance-dies-the-number-of-data-stored-in-elasticsearch-increases/209320>\
**Category:** Logstash\
**Created:** [November 25, 2019, 2:08pm UTC](https://discuss.elastic.co/t/each-time-logstash-instance-dies-the-number-of-data-stored-in-elasticsearch-increases/209320 "2019-11-25T14:08:32Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![securecode99](https://avatars.discourse-cdn.com/v4/letter/s/5e9695/32.png) [@securecode99](https://discuss.elastic.co/u/securecode99)\
**Post date:** [November 25, 2019, 2:08pm UTC](https://discuss.elastic.co/t/each-time-logstash-instance-dies-the-number-of-data-stored-in-elasticsearch-increases/209320/1 "2019-11-25T14:08:33Z")

</div>

I'm using ELK(Elasticsearch, Logstash, Kibana, Beat) with a Kafka in front of Logstash.

However, I noticed something unusual during the failover test of Logstash.

The test was carried out by suddenly killing the one logstash instance while loading 10,000 logs of files from filebeat to kafka. (logstash did not use any filter, only input, output.)

The result of the test was that each time there was a logstash problem, the number of data loaded on elasticsearch increased by approximately 7000 to 9000 more.

In relation to this test, I conducted tests with changes in partition and replication factor of kafka-topic, and found that only the kafka-topic index assigned to the logstash process in question produce duplicate data.

Why does the number of data increase when logstash instance dies?

Please help me. Thank you.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [November 25, 2019, 3:15pm UTC](https://discuss.elastic.co/t/each-time-logstash-instance-dies-the-number-of-data-stored-in-elasticsearch-increases/209320/2 "2019-11-25T15:15:55Z")

</div>

Have you tried enabling [persistent queues](https://www.elastic.co/guide/en/logstash/current/persistent-queues.html) and disabling [auto\_commit](https://www.elastic.co/guide/en/logstash/current/plugins-inputs-kafka.html#plugins-inputs-kafka-enable_auto_commit)?

---

<div class="post-metadata">

**Author:** ![securecode99](https://avatars.discourse-cdn.com/v4/letter/s/5e9695/32.png) [@securecode99](https://discuss.elastic.co/u/securecode99)\
**Post date:** [December 2, 2019, 9:38am UTC](https://discuss.elastic.co/t/each-time-logstash-instance-dies-the-number-of-data-stored-in-elasticsearch-increases/209320/3 "2019-12-02T09:38:34Z")

</div>

Hello, we've done a test based on your answers.

First, auto\_commit was false in the kafka, and secondly, the logstash enabled persistent\_queues.

As a result, these options seemed to have no effect on duplicate\_data.

Are there other option values to look at?

Thank you for your answer.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [December 30, 2019, 9:38am UTC](https://discuss.elastic.co/t/each-time-logstash-instance-dies-the-number-of-data-stored-in-elasticsearch-increases/209320/4 "2019-12-30T09:38:35Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.