I'm having an issue with persisted queues in logstash. After about a day of running, the persisted queue fills up to the max bytes and old page files are not being removed. I have inputs set up for beats and syslog and an Elasticsearch output with an index rate of around 700/s.
I can confirm that Elasticsearch is indexing properly, no data is missing, and everything is up to date.
Here is my logstash.yml:
path:
data: "/var/lib/logstash"
config: "/etc/logstash/conf.d"
logs: "/var/log/logstash"
queue:
type: persisted
page_capacity: 250mb
max_bytes: 16gb
Yet the persisted queue at /usr/share/logstash/data/queue continues to fill up with page files (the first always being page.0) and none are ever removed. Eventually the queue fills up and logstash has to be stopped and the queue cleared manually.