Find the Slowest Filebeat

Elastic Stack Beats
1

Hello,
I am new to ELK stack. I have several Filebeats configured to send the logs to Logstash and also enabled Filebeat to send the monitoring data to Elastic search. I have a downstream process that needs to process the events until the slowest sender's last published timestamp to avoid unnecessary errors. The downstream process needs to pause the processing if any of the Filebeats is down or is lagging behind.

For e.g. if there are 3 Filebeats and the last time stamps they successfully sent and got acknowledgement from Logstash are 2020-05-20 10:00:00.001, 2020-05-20 10:00:05.565 and 2020-05-20 10:00:05.575 respectively. Then I would like the downstream process to process the events only until 2020-05-20 10:00:00.001 and pause since that is the last published and acknowledged timestamp from slowest Filebeat (Filebeat1).

I was planning to write logic in the downstream process to query Elastic search and use the heartbeat data to find the slowest Filebeat.

  1. Is there a way I could figure out the slowest Filebeat using the following fields for each file in the harvester from the monitoring heartbeat?
    last_event_published_time 
    last_event_timestamp
    read_offset 
    size
  1. Does last_event_published_time mean the last time Filebeat published an event and got an acknowledgement from Logstash and does last_event_timestamp mean last time it read an event from log file?

Thanks in advance for all the help,
Suvelee

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.