How can I monitor the queue of Filebeat

Elastic Stack Beats
1

I'm currently using the default setting for filebeat, and also sends the monitoring data to an Elasticsearch cluster.

My workflow for logs is Filebeat -> Logstash -> Elasticsearch

Just wondering how can we monitor the internal queue of Filebeat? From my understanding when Filebeat doesn't receive ack from Logstash it will queue up the data.
But how to tell if the queue is full and how to get alerts from it?

The monitoring dashboard on Kibana doesn't seem to reflect the relevant data.

2

It is possible to gather metrics from both beats and logstash and send its metrics to Elasticsearch, you mention that Kibana did not show the metrics you are after, but is that with this enabled?

You can see a quick howto here on how to do it for both:

3

Yes I have all these enabled. But which metrics refer to the Filebeat queue status?

E.g.
For Filebeat my setting queue.mem.events = 4096
But from the metrics I don't see anything like x/4096
I don't see the number of events in the queue.

I simulated a situation when Logstash is down, and I can see metrics about Fail Rates.
But how can I know if the Filebeat's internal queue is full? In my production environment I have zero tolerance on losing events, so I need to know the status of Filebeat's queue and take action immediately

p1
p11873×828 112 KB

2 Likes
4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.