I want to open an enhancement request for this but wanted to get feedback here first...
When using persistent queueing in Logstash access to near real time data is not available until queued data is worked off. The prevents access to live data that is being sent to logstash and ultimately prevents knowledge of metrics, logs and security information until the queued data is sent.
Example Scenario: There are numerous servers at a site which are sending metrics, windows events and application logs to an instance of logstash at the site. All is working fine on Friday when everyone goes home. The connection from the logstash instance to the Elastic cluster goes down late Friday night so the Logstash instance writes data to it's persistent queue all weekend. On Monday the workers return and find this connection issue and resolve it. Logstash detects the correction and starts working off the queue, but there is hours of data to work off. The workers bring up the dashboards they normally use to check that status of the site, but cannot see any data because it's so far behind. They have no way of knowing the current status of the servers at the site even though the servers are sending current data to logstash. They have to wait until all data in the persistent queue is worked off before they can see the near real time status of the servers.
I would like an option added to logstash to allow prioritizing live data over queued data. The queued data should still be sent but possibly over a second pipeline or mixed in with the current data. The affect should be that as soon as connectivity from logstash to elastic is restored logstash should start sending newly ingest data directly to elastic instead of continuing to add it to the queue. At the same time it should also start working the data off the queue. The affect will be users will immediately have access to current data and will eventually have access to all persisted data.
I love the idea of persistent queues but the current implementation that prevents access to current data upon resolving connectivity issues prevents me from using it.