Persisted queue checkpoint and acknowledged inputs

Elastic Stack Logstash
1

Hi,
I recently had to dig in our stack to identify some missing events, and one of the candidate explanations sits in this logstash doc section :

By default, PQ will force write data to disk at every 1024 received input events to reduce potential data loss upon OS level crash. Note that application and process level crashes or interrupted shutdowns should never result in data loss, regardless of the queue.checkpoint.writes setting. If you want to protect against potential data loss for OS level crash you can set PQ for maximum durability using the queue.checkpoint.writes:1 setting, but note that this will result in a significant IO performance reduction.

So I understand that, if I want to ensure 100% durability, 100% prevented against unexpected interruption, I have to set this queue.checkpoint.writes:1 option.

Before embark into this path, as we use only "acknowledged inputs" (jdbc, file, beats...). I wonder if this kind of input plugin are able to force a queue checkpoint just before the aknowledgment. In this case, in case of interruption, all events for this input since the last commited aknowledgement (including the max 1024 events only in OS buffers) would be guaranteed to be replayed.

Am I wrong ?

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.