I have the following setup: [tcp, http, file, udp] ---> logstash ---> [custom output plugin]
And a few questions on persistent queues.
As per the limitions do the tcp and udp inputs enter the persistent queue or bypass it completely? Does "not protected against data loss" just mean that the data source won't receive an ACK and thus doesn't know if it should "release" the event if logstash has a failure?
For output, does logstash only remove from the persistent queue when it receives an ACK from the data destination?
Is there any guidance on how to develop an output plugin to be compatible with Persistent queues? I assume there must be some feedback from the output plugin to indicate that the data destination has successfully received the data, therefore allowing logstash to pop from the queue?
Are output plugins such as udp incompatible with persistent queues due to the lack of guaranteed delivery / no ACK?
Very astute questions! I'll attempt to answer in-line:
When the PQ is enabled on a pipeline, all inputs put items into the PQ (whether or not they can provide at-least-once delivery semantics).
When a pipeline consistently is processing events slower than its inputs are putting them into the queue, the queue can fill up. When the queue reaches the configured maximum capacity, attempts by inputs to put things into the queue will block until there is sufficient available capacity. Unfortunately, with inputs like TCP and UDP, the input blocking simply means that it is no longer reading the bytes from the underlying socket; the OS will continue to accept input (for TCP, this may be ACK'd at the protocol level). These buffers, bytes that have been received by the OS but have not been read by the TCP/UDP input, are at risk. The sender has handed off responsibility, but Logstash hasn't yet actually accepted responsibility.
Each Output Plugin is implemented differently, but in general, outputs retry their batches of input until success. If the Dead Letter Queue is enabled, failures can be flushed to a separate disk-based queue for later analysis and replay.
Mostly: just keep trying; don't return until the plugin has either succeeded or has handed off the events to the DLQ.
There is still some benefit, since the PQ provides a buffer that is resilient across process restarts. This buffer is often big enough to handle momentary spikes in traffic, accepting the UDP packets off of the more fragile buffers and getting them onto disk.
If you're just looking into PQ's, make sure you start with at least Logstash 6.3.0 -- we had several bugs get fixed in that release.
Each Output Plugin is implemented differently, but in general, outputs retry their batches of input until success. If the Dead Letter Queue is enabled, failures can be flushed to a separate disk-based queue for later analysis and replay.
Ok, so when the data is being processed by my Output plugin it has already been popped from the PQ? If the output plugin is unable to deliver it, or Logstash was to reboot/crash at this point then the data will be lost? (Assuming the Output plugin doens't have some kind of buffering mechanism for failed sends)
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.