# Persistent Queues - invitation for beta testing! **URL:** \ **Category:** Logstash\ **Created:** [January 24, 2017, 9:56pm UTC](https://discuss.elastic.co/t/persistent-queues-invitation-for-beta-testing/72705 "2017-01-24T21:56:01Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![alvin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alvin/32/5607_2.png) [@alvin](https://discuss.elastic.co/u/alvin)\ **Post date:** [January 24, 2017, 9:56pm UTC](https://discuss.elastic.co/t/persistent-queues-invitation-for-beta-testing/72705/1 "2017-01-24T21:56:01Z")
Logstash 5.1 introduced an exciting new feature called Persistent Queues (PQs), which provides two main benefits to Logstash users: 1. **Message durability** - protection against in-flight data loss. All incoming events will be persisted to disk, so Logstash will be durable across instance failures. 2. **Simplified ingest architectures** - adaptive buffering now handles ingest spikes natively in Logstash without the need for a separate queuing layer. The buffering can be capped by max byte size or number of events. For logging use cases, basic Filebeat/Winlogbeat → Logstash → Elasticsearch ingestion flows can be used with the comfort of durability. Logstash, by default, uses a fixed, in-memory queue between pipeline stages to help facilitate dataflow. By enabling PQs, this internal queue now becomes both disk-based and variable length. Although persisting to disk often comes with a performance cost, our initial benchmarks have shown that the performance impact of enabling PQs is essentially negligible for most real-world use cases. This is more great news for our users! Please note that this feature is **currently in beta** and should be deployed in production at your own risk. Users who try out PQs during the beta cycle and **[report legitimate bugs or feedback in Github](https://github.com/elastic/logstash/issues/new)** will receive a **special Elastic gift package** as our thank you. We kindly invite everyone to try it out! For additional details on the feature and how to enable it, please consult the **[Persistent Queues documentation](https://www.elastic.co/guide/en/logstash/current/persistent-queues.html)**. ---
**Author:** ![alvin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alvin/32/5607_2.png) [@alvin](https://discuss.elastic.co/u/alvin)\ **Post date:** [January 24, 2017, 9:57pm UTC](https://discuss.elastic.co/t/persistent-queues-invitation-for-beta-testing/72705/2 "2017-01-24T21:57:09Z")
---
**Author:** ![fortikeco](https://avatars.discourse-cdn.com/v4/letter/f/a587f6/32.png) [@fortikeco](https://discuss.elastic.co/u/fortikeco)\ **Post date:** [February 16, 2017, 1:32pm UTC](https://discuss.elastic.co/t/persistent-queues-invitation-for-beta-testing/72705/3 "2017-02-16T13:32:46Z")
Does that mean that there is still only 1 event-pipeline? The reason I ask, is because in the setup you describe (Filebeat -\> logstash -\> ES) implies that there could be multiple logging types being sent to logstash. When Logstash processes the messages concurrently on 1 pipeline, every message and its contents needs to be checked for its type, which not only costs resources but also ends up in big configuration. ---
**Author:** ![alvin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alvin/32/5607_2.png) [@alvin](https://discuss.elastic.co/u/alvin)\ **Post date:** [February 16, 2017, 8:51pm UTC](https://discuss.elastic.co/t/persistent-queues-invitation-for-beta-testing/72705/4 "2017-02-16T20:51:03Z")
Yes, only one pipeline today, though you can still certainly use stream identity and conditionals in Logstash to handle different types of logs from Filebeat. Multi-pipeline support is on the roadmap: [https://github.com/elastic/logstash/issues/6521](https://github.com/elastic/logstash/issues/6521) ---
**Author:** ![fortikeco](https://avatars.discourse-cdn.com/v4/letter/f/a587f6/32.png) [@fortikeco](https://discuss.elastic.co/u/fortikeco)\ **Post date:** [February 17, 2017, 8:42am UTC](https://discuss.elastic.co/t/persistent-queues-invitation-for-beta-testing/72705/5 "2017-02-17T08:42:04Z")
ok thanks for the information! ---
**Author:** ![fizwit](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/fizwit/32/6421_2.png) [@fizwit](https://discuss.elastic.co/u/fizwit)\ **Post date:** [March 13, 2017, 4:32pm UTC](https://discuss.elastic.co/t/persistent-queues-invitation-for-beta-testing/72705/6 "2017-03-13T16:32:20Z")
Is there any acknowledgment between Logstash and ES that an event has been received and indexed by ES? How do Persistent Queues work with log files? ---
**Author:** ![alvin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alvin/32/5607_2.png) [@alvin](https://discuss.elastic.co/u/alvin)\ **Post date:** [March 16, 2017, 7:39am UTC](https://discuss.elastic.co/t/persistent-queues-invitation-for-beta-testing/72705/7 "2017-03-16T07:39:56Z")
@fizwit yes, it acks. Starting with the 5.3 release, Logstash will guarantee at-least-once delivery to Elasticsearch with Persistent Queues (PQs) enabled. For most logging use cases, you should use Filebeat -\> Logstash -\> Elasticsearch to yield at-least-once guarantees and ensure security across the delivery chain. Hope this helps. ---
**Author:** ![alvin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alvin/32/5607_2.png) [@alvin](https://discuss.elastic.co/u/alvin)\ **Post date:** [April 13, 2017, 7:40am UTC](https://discuss.elastic.co/t/persistent-queues-invitation-for-beta-testing/72705/8 "2017-04-13T07:40:29Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.