Logstash 5.1 introduced an exciting new feature called Persistent Queues (PQs), which provides two main benefits to Logstash users:
Message durability - protection against in-flight data loss. All incoming events will be persisted to disk, so Logstash will be durable across instance failures.
Simplified ingest architectures - adaptive buffering now handles ingest spikes natively in Logstash without the need for a separate queuing layer. The buffering can be capped by max byte size or number of events. For logging use cases, basic Filebeat/Winlogbeat → Logstash → Elasticsearch ingestion flows can be used with the comfort of durability.
Logstash, by default, uses a fixed, in-memory queue between pipeline stages to help facilitate dataflow. By enabling PQs, this internal queue now becomes both disk-based and variable length. Although persisting to disk often comes with a performance cost, our initial benchmarks have shown that the performance impact of enabling PQs is essentially negligible for most real-world use cases. This is more great news for our users!
Please note that this feature is currently in beta and should be deployed in production at your own risk. Users who try out PQs during the beta cycle and report legitimate bugs or feedback in Github will receive a special Elastic gift package as our thank you. We kindly invite everyone to try it out!
For additional details on the feature and how to enable it, please consult the Persistent Queues documentation.