Thanks @guyboertje for the detailed description. But i have been observing the behavior of the queues by creating a small script which gives out the number of events processed by the pipeline. Whenever the persistent queues are enabled, and if there is a sudden spike in the incoming logs (around 1 million in 5 minutes), the pipeline stops processing the events(which is expected according to documentation). But the problem is that when the input events reduce the pipeline is not getting back up to process the events and the logs in redis keep on increasing.
The above image shows the sudden spike in output events to elasticsearch and after that the pipeline does not start processing the logs from the redis queue.
I see this behaviour only when persisted queues are enabled.
Above image is the behaviour when the persistent queues are disabled.