Feature: max_age of events in persistent queue / time based cleanup

The persistent queue can be configured to hold a specific number of events, and it can be limited in size (see persistent queue doc) , the parameters are called queue.max_events and queue.max_bytes. Due to regulations, we have to ensure that we do not store events older than x minutes. It would be great to have an age-based cleanup, and a parameter queue.max_age. It would be good enough to use the timestamp of the event when it entered the queue.

One workaround idea that a collegue came up with is to delete the queue pages on disk after x minutes. But we didn't test how logstash would take that.

The information of how much time the event spent in logstash seems to exist in duration_in_millis , but I'm not sure when exactly that starts and stops.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.