Does Filebeat take in logs missed when it is stopped?

Elastic Stack Beats
1

I have filebeat 7.8.1 streaming logs to my elasticsearch via this setup

Application -> Logfile -> Filebeat -> Logstash -> Elasticsearch

The application team is concerned if Filebeat is down, it will cause logs to be missed,
Will Filebeat be able to detect which logs are sent during it's downtime and automatically send them to logstash when it is turned back on?

2

Filebeat tracks the position of the harvested files, if you stop filebeat and start again later, it will read from the last know position when it was running.

If the log is rotated while filebeat was in a stop state it will not be able to send the events that happened between that time.

But you can always reparse the logs if needed.

3

I see, thank you very much for the quick response

4

Hi leandrojmp,

The application team tried to delete the log file and each time the log file is deleted, in the directory /data/registry/filebeat/log.json keeps growing in size

Would this be a concern if this log.json keeps growing?

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.