Persisted queue never clearing

I'm having an issue with persisted queues in logstash. After about a day of running, the persisted queue fills up to the max bytes and old page files are not being removed. I have inputs set up for beats and syslog and an Elasticsearch output with an index rate of around 700/s.

I can confirm that Elasticsearch is indexing properly, no data is missing, and everything is up to date.

Here is my logstash.yml:

  data: "/var/lib/logstash"
  config: "/etc/logstash/conf.d"
  logs: "/var/log/logstash"
  type: persisted
  page_capacity: 250mb
  max_bytes: 16gb

Yet the persisted queue at /usr/share/logstash/data/queue continues to fill up with page files (the first always being page.0) and none are ever removed. Eventually the queue fills up and logstash has to be stopped and the queue cleared manually.

1 Like

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.