hi everyone,
how does filebeat handle the case where a different log file is received every hour from a cloud system and sometimes the received files have no data (0kb). ? Do these files with no data cause any issues for filebeat?
It would be helpful to know a few more details like naming convention of the files, any rotation that might be going on, etc. but based on what you've posted Filebeat should not have any issues.
When a new file is created in the path being watched by Filebeat, Filebeat will detect that file within a certain period of time (determined by the scan_frequency setting, default value = 10 seconds). and open a new harvester for it. That harvester will try to read the file, line by line, until it reaches EOF. Once it reaches EOF, the harvester will hold on the file handle for a certain period of time (determined by the close_inactive setting, default value = 5 minutes), in case more lines show up. If no new lines show up in that period of time, the file handle is closed. If new lines then get added to the file, a new harvester is started. Filebeat always keeps track of how far into the log file it has read so it can start from that position onwards.
So, in your case, if a 0-byte file shows up, a harvester will get started for it. If the file remains 0 bytes for more than the close_inactive duration, the harvester will release the file handle and close itself.
Hope that helps,
Shaunak
Thank you so much! this is exactly what i was after!!! 
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.