In our log system, our local log files is automatically rotated, for example, every hour or every midnight, the current log file (e.g., myapp.log) will be renamed (maybe be also compressed) to myapp.log.day1.gz, and start a new log file with the same name myapp.log
I am wondering whether Filebeat can handle this log rotation correctly? from reading the documentation, when the existing log file is renamed, the harvester will hold on the file descriptor and can probably continue to process the remaining log lines. However, can it handle the new log lines written to the new created file with the same file name? or the prospector will create a new harvester for this new log file with the same log file name as the original one?
In addition to handling log rotation under normal condition, can it handle log rotation under the failure mode? for example:
Filebeat crashed for some reason. Since the time it crashes the original myapp.log file was already renamed to my app.log.day2.
FileBeat is configured to output to kafka, and kafka somehow became unavailable. The log files are being rotated while kafka became unavailable.
Any pointer is appreciated!