I've seen some behaviour with persistent queues that surprises me.I noticed this when throughput was capped on a logstash node when the EBS Burst Balance was exceeded for the volume of the logstash node in question. Here is a screenshot from cloudwatch illustrating the behaviour:
Given that the persistent queue length is usually either very low or zero it makes me wonder whether every inbound message is being written to and read back from disk before being indexed to elasticsearch?
The persistent queue overview that correlates with the above screenshot can be seen here:
So, can someone clarify for me what my expectations should be when using persistent queues on EBS volumes ? Why does persistent queueing eat so much burst quota even when the node isn't falling behind particularly badly?