We are using Persistent Queues, and I would like to calculate the time an event is in queue before actually being sent through the pipeline, but in order to do so, I need to add a timestamp in the input section of the pipeline.
I have tried just copying @timestamp, and that works perfectly fine for TCP and UDP inputs, as the timestamp is added when the event is received. However, for Beats agent input, it seems the @timestamp field is written by the Beats agent itself, so that ruins the entire idea of calculating queue time.
How can we add a current timestamp to events when they hit the Logstash input, but before the event hits the queue?
Would appreciate any thoughts.