` is at 100MB and gradually increasing till 500MB. Does this mean that the data is not being sent over to ES? What does this file contain and when will it be again set to 0MB?
Thanks
---
### Author:  [@guyboertje](https://discuss.elastic.co/u/guyboertje)
#### Post date: [February 1, 2017, 5:22pm UTC](https://discuss.elastic.co/t/persistent-queue-configuration-question/73408/7 "2017-02-01T17:22:03Z")
How much test data are you putting into redis?
Previously, did you see events in ES and was Redis empty?
---
### Author:  [@ash007](https://discuss.elastic.co/u/ash007)
#### Post date: [February 1, 2017, 5:55pm UTC](https://discuss.elastic.co/t/persistent-queue-configuration-question/73408/8 "2017-02-01T17:55:16Z")
@guyboertje, I am sorry, the data in redis is not finite. I am feeding live data to the cluster and currently its around 500-1000 events/sec
---
### Author:  [@ash007](https://discuss.elastic.co/u/ash007)
#### Post date: [February 2, 2017, 7:23pm UTC](https://discuss.elastic.co/t/persistent-queue-configuration-question/73408/9 "2017-02-02T19:23:10Z")
Hello @guyboertje, I have another query. Is there any way we can confirm the existence of persistence queues?
Because i tried to get stats on the pipeline by using `curl -XGET 'localhost:9600/_node/stats/pipeline?pretty'` and i see that in the result there is no block showing the stats of persistent queues
> ` "reloads" : { "last_error" : null, "successes" : 0, "last_success_timestamp" : null, "last_failure_timestamp" : null, "failures" : 0 } }`
> as mentioned [in the docs.](https://www.elastic.co/guide/en/logstash/current/node-stats-api.html#pipeline-stats)
> Thanks.
---
### Author:  [@guyboertje](https://discuss.elastic.co/u/guyboertje)
#### Post date: [February 3, 2017, 9:49am UTC](https://discuss.elastic.co/t/persistent-queue-configuration-question/73408/10 "2017-02-03T09:49:36Z")
Metrics for the persistent queue was not shipped in the initial release. It is coming soon.
---
### Author:  [@ash007](https://discuss.elastic.co/u/ash007)
#### Post date: [February 9, 2017, 9:57pm UTC](https://discuss.elastic.co/t/persistent-queue-configuration-question/73408/11 "2017-02-09T21:57:35Z")
@guyboertje, I see that whenever the input load increases, multiple checkpoint files are created. When the corresponding page file is processed the checkpoint file needs to be deleted automatically. But my queue folder is filled with multiple files.
Example:
This is resulting in stalling the logstash shutdown or start process.
---
### Author:  [@guyboertje](https://discuss.elastic.co/u/guyboertje)
#### Post date: [February 10, 2017, 10:03am UTC](https://discuss.elastic.co/t/persistent-queue-configuration-question/73408/12 "2017-02-10T10:03:13Z")
When the inflow rate of events is higher than the output, as you now know, page and checkpoint files build up.
If a worker thread is slow or very very slow (stuck) it will not have acked its batch so the page and checkpoint that the batch was read from can't be closed and deleted. But other workers will read and ack later pages. In this specific case when a later page is fully acked we only delete the page file and not its checkpoint. This means the PQ has a complete 'breadcrumb' trail from the checkpoint.head back to the unacked page and checkpoint. This allows us to tell on startup if the PQ is corrupted should pages and checkpoints be deleted by some other means.
If the slow worker does eventually loop back and ack its batch then we remove that page and checkpoint and clean the intermediate checkpoint files.
---
### Author:  [@ash007](https://discuss.elastic.co/u/ash007)
#### Post date: [February 15, 2017, 2:51pm UTC](https://discuss.elastic.co/t/persistent-queue-configuration-question/73408/13 "2017-02-15T14:51:53Z")
Thanks @guyboertje for the detailed description. But i have been observing the behavior of the queues by creating a small script which gives out the number of events processed by the pipeline. Whenever the persistent queues are enabled, and if there is a sudden spike in the incoming logs (around 1 million in 5 minutes), the pipeline stops processing the events(which is expected according to documentation). But the problem is that when the input events reduce the pipeline is not getting back up to process the events and the logs in redis keep on increasing.
The above image shows the sudden spike in output events to elasticsearch and after that the pipeline does not start processing the logs from the redis queue.
I see this behaviour only when persisted queues are enabled.
Above image is the behaviour when the persistent queues are disabled.
---
### Author:  [@guyboertje](https://discuss.elastic.co/u/guyboertje)
#### Post date: [February 16, 2017, 4:20pm UTC](https://discuss.elastic.co/t/persistent-queue-configuration-question/73408/14 "2017-02-16T16:20:15Z")
@ash007
This seems to be a bug. Please file a bug report at [https://github.com/elastic/logstash/issues](https://github.com/elastic/logstash/issues)
Make sure you give us extremely detailed instructions of what you did, your versions, configs and sample data.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [March 16, 2017, 4:20pm UTC](https://discuss.elastic.co/t/persistent-queue-configuration-question/73408/15 "2017-03-16T16:20:40Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.