# Can Logstash lose data?

**URL:** <https://discuss.elastic.co/t/can-logstash-lose-data/205535>\
**Category:** Logstash\
**Created:** [October 28, 2019, 6:32pm UTC](https://discuss.elastic.co/t/can-logstash-lose-data/205535 "2019-10-28T18:32:51Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![amitavmohanty01](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/amitavmohanty01/32/58017_2.png) [@amitavmohanty01](https://discuss.elastic.co/u/amitavmohanty01)\
**Post date:** [October 28, 2019, 6:32pm UTC](https://discuss.elastic.co/t/can-logstash-lose-data/205535/1 "2019-10-28T18:32:51Z")

</div>

If I understand correctly, Logstash has two thread pools: input (IN), processing and output combines (OUT). When Logstash is getting input from Kafka, at what point does it send an ACK to Kafka? If the ACK is sent when the IN buffer moves data moves buffer to OUT buffer, then there is a chance that there can be a loss of data if the process is restarted and there is some data in the OUT buffer which is not sent to Elasticsearch. However, if the ACK is sent after the data is sent to Elasticsearch, then process restart will always start where it left off.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [October 28, 2019, 7:40pm UTC](https://discuss.elastic.co/t/can-logstash-lose-data/205535/2 "2019-10-28T19:40:37Z")

</div>

I believe the input acks the receipt of data from Kafka as soon as it receives it. You can use [persistent queues](https://www.elastic.co/guide/en/logstash/current/persistent-queues.html) to avoid data loss.

---

<div class="post-metadata">

**Author:** ![amitavmohanty01](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/amitavmohanty01/32/58017_2.png) [@amitavmohanty01](https://discuss.elastic.co/u/amitavmohanty01)\
**Post date:** [October 29, 2019, 4:49am UTC](https://discuss.elastic.co/t/can-logstash-lose-data/205535/3 "2019-10-29T04:49:19Z")

</div>

Using persistent queues to avoid data loss is costly because of the associated storage. It is costly in terms of both time and money.

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [October 29, 2019, 6:22am UTC](https://discuss.elastic.co/t/can-logstash-lose-data/205535/4 "2019-10-29T06:22:38Z")

</div>

There is an [open issue](https://github.com/elastic/logstash/issues/8514) for making Logstash capable to running in a stateless mode where the input is not acknowledged until the data has been written successfully to the outputs. This would remove the need for an internal persistent queue, but does not appear to be worked on. So for now a persistent queue is your best bet if you want to avoid data loss.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [November 26, 2019, 6:36am UTC](https://discuss.elastic.co/t/can-logstash-lose-data/205535/5 "2019-11-26T06:36:29Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.