The persistent queue can be configured to hold a specific number of events, and it can be limited in size (see persistent queue doc) , the parameters are called queue.max_events
and queue.max_bytes
. Due to regulations, we have to ensure that we do not store events older than x minutes. It would be great to have an age-based cleanup, and a parameter queue.max_age
. It would be good enough to use the timestamp of the event when it entered the queue.
One workaround idea that a collegue came up with is to delete the queue pages on disk after x minutes. But we didn't test how logstash would take that.
The information of how much time the event spent in logstash seems to exist in duration_in_millis
, but I'm not sure when exactly that starts and stops.