The Logstash Metric filter and reality

Elastic Stack Logstash
1

I have been monitoring log flow by just using a simple dashboard with lots of histogram charts showing the flow of incoming logs by tags. This way I can see if a certain Logstash input has stopped or is experiencing heavy load.

Now I'm doing a simple experiment with the metric filter hoping to get the same results. The difference is that if I work against the real logs the timestamps are corrected so when I get a huge spike in traffic and logs start to pile up in the queue. Using the metric filter it is only measuring the amount Logstash can process but not the real influx of logs.

Is there any way to have the metric filter use the timestamps?

2

Nope, that's not what it's been designed for unfortunately.
Why not just monitor the queue?

3

Depends on what I want to monitor.

My pipeline is like this

Incoming sources -> Single Logstash Process, just input and output -> Multiple RabbitMQ Queues- > Multiple Logstash processes, one for each Queue, lots of filters

I can use the metric filter on the first Logstash instance to monitor basically each queue. That will be accurate mostly

But if I want to use metrics on data that will be parsed from the queues those metrics will only be valid as long as nothing builds up in the queue.