# Logstash Data Persistency

**URL:** <https://discuss.elastic.co/t/logstash-data-persistency/263948>\
**Category:** Logstash\
**Created:** [February 11, 2021, 4:21am UTC](https://discuss.elastic.co/t/logstash-data-persistency/263948 "2021-02-11T04:21:52Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![leventyalcin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leventyalcin/32/83757_2.png) [@leventyalcin](https://discuss.elastic.co/u/leventyalcin)\
**Post date:** [February 11, 2021, 4:21am UTC](https://discuss.elastic.co/t/logstash-data-persistency/263948/1 "2021-02-11T04:21:52Z")

</div>

Hi,

As you might know, in-flight queue is hold in memory by Logstash.  
I'm trying to enable data persistency with a persistent queue. However, restarting logstash or the instance wipes out the data.

When I saw that I tried many ways but the result was the same. My last test was

1. Add sleep filter to the pipeline and sleep 1 second for every 1 event
2. Sent 120 message to the Logstash from journalbeat to logstash
3. When the TCP packages arrive to the logstash instance, disable firewall rule for elasticsearch. So, logstash cannot deliver events.

At this point, I was cURL-ing logstash's API and I can clearly see the messages were received, processed by all the filters and waiting at sleep filter. It can be seen in here [Logstash Data Persistency · GitHub](https://gist.github.com/leventyalcin/cdc809c19dc38815aef9a4548a503e69#file-logstash-api-json-L241-L249)

Even when I have the config for queue.checkpoint.writes: 10

> <https://gist.github.com/leventyalcin/cdc809c19dc38815aef9a4548a503e69#file-logstash-yml-L6>

, the queue always seems 0 in here [Logstash Data Persistency · GitHub](https://gist.github.com/leventyalcin/cdc809c19dc38815aef9a4548a503e69#file-logstash-api-json-L355-L357) and in here [Logstash Data Persistency · GitHub](https://gist.github.com/leventyalcin/cdc809c19dc38815aef9a4548a503e69#file-logstash-api-json-L311-L328)

When I restart logstash at that point, all the events vanish and the size of the queue seems 0 on logstash API and those events never come to ElasticSearch.

It's either I'm missing something or data persistency is not working on logstash 7.10.2-1 / 7.11.0-1

I'd highly appreciate if you see anything I'm missing and let me know.

Thanks,  
Levent.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 11, 2021, 4:22am UTC](https://discuss.elastic.co/t/logstash-data-persistency/263948/2 "2021-03-11T04:22:42Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.