I am new to ELK stack. I have several Filebeats configured to send the logs to Logstash and also enabled Filebeat to send the monitoring data to Elastic search. I have a downstream process that needs to process the events until the slowest sender's last published timestamp to avoid unnecessary errors. The downstream process needs to pause the processing if any of the Filebeats is down or is lagging behind.
For e.g. if there are 3 Filebeats and the last time stamps they successfully sent and got acknowledgement from Logstash are 2020-05-20 10:00:00.001, 2020-05-20 10:00:05.565 and 2020-05-20 10:00:05.575 respectively. Then I would like the downstream process to process the events only until 2020-05-20 10:00:00.001 and pause since that is the last published and acknowledged timestamp from slowest Filebeat (Filebeat1).
I was planning to write logic in the downstream process to query Elastic search and use the heartbeat data to find the slowest Filebeat.
- Is there a way I could figure out the slowest Filebeat using the following fields for each file in the harvester from the monitoring heartbeat?
last_event_published_time last_event_timestamp read_offset size
last_event_published_timemean the last time Filebeat published an event and got an acknowledgement from Logstash and does
last_event_timestampmean last time it read an event from log file?
Thanks in advance for all the help,