Logstash Data Persistency

Elastic Stack Logstash
1

Hi,

As you might know, in-flight queue is hold in memory by Logstash.
I'm trying to enable data persistency with a persistent queue. However, restarting logstash or the instance wipes out the data.

When I saw that I tried many ways but the result was the same. My last test was

  1. Add sleep filter to the pipeline and sleep 1 second for every 1 event
  2. Sent 120 message to the Logstash from journalbeat to logstash
  3. When the TCP packages arrive to the logstash instance, disable firewall rule for elasticsearch. So, logstash cannot deliver events.

At this point, I was cURL-ing logstash's API and I can clearly see the messages were received, processed by all the filters and waiting at sleep filter. It can be seen in here Logstash Data Persistency · GitHub

Even when I have the config for queue.checkpoint.writes: 10

, the queue always seems 0 in here Logstash Data Persistency · GitHub and in here Logstash Data Persistency · GitHub

When I restart logstash at that point, all the events vanish and the size of the queue seems 0 on logstash API and those events never come to ElasticSearch.

It's either I'm missing something or data persistency is not working on logstash 7.10.2-1 / 7.11.0-1

I'd highly appreciate if you see anything I'm missing and let me know.

Thanks,
Levent.

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.