Logstash persistent queue feature

[tomx1]

I was just reading this article of the new Logstash persistent queue feature: Persistent queues (PQ) | Logstash Reference [8.11] | Elastic

I think it's great that Logstash is now offering that feature, but theres a part in the description which is a bit concerning to me:

An event is recorded as ACKed in the checkpoint file if the event is successfully sent to the last output stage in the pipeline; Logstash does not wait for the output to acknowledge delivery.

So if Logstash is not waiting for the output filter to acknowledge delivery, what if - for example - the elasticsearch output of logstash is not able to deliver the event to elasticsearch due an network error?

[jordansissel]

This documentation looks confusing and possibly misleading.

When I read the code, I see the following:

• Outputs receive events
• Then the current work batch (of events) is acknowledged in the queue.

In the code, we ack to the queue by "closing" the work unit (called a batch, internally). This work contains a bunch of events to operate on.

Citations:

• output receiving followed by close_batch: https://github.com/elastic/logstash/blob/master/logstash-core/lib/logstash/pipeline.rb#L288-L289
• close_batch calls batch.close: https://github.com/elastic/logstash/blob/master/logstash-core/lib/logstash/util/wrapped_acked_queue.rb#L168
• batch.close acknowledges to the queue: https://github.com/elastic/logstash/blob/master/logstash-core-queue-jruby/src/main/java/org/logstash/ackedqueue/ext/JrubyAckedBatchExtLibrary.java#L80 which calls https://github.com/elastic/logstash/blob/master/logstash-core/src/main/java/org/logstash/ackedqueue/Batch.java#L26

---

The way I read it, and from what I remember during the design/code review of this feature, is that the output plugin will be given the events, and once that happens, we acknowledge.

In the case of Elasticsearch output having a network error, the Elasticsearch output will retry most kinds of errors. This retry mechanism will block until it is successful, which means that the pipeline will not have the opportunity to ack events in the queue until the output plugin's receive or multi_receive call has completed.

In summary, assuming my recollection of the design is accurate: An event will not be acked in the persistent queue until all outputs have finished receiving the event, so by default, most outputs will deliver their events downstream (to elasticsearch, etc) before the events are acked in the queue.

[tomx1]

Thanks jordan!