We are using filebeat on windows 7 to send logs to kafka. Filebeat version used is 5.2.2.
Filebeat.yml has config similar to following
- paths: - somepath\*.txt input_type: log document_type: abc ignore_older: 12h close_inactive: 1m
In the filebeat logs, there is a line for closing a file, which is immediately followed by line saying that a harvester is started for the same file. Following set of logs are repeated until the time file is deleted. Basically a harvester is repeatedly opened and closed for the inactive file.
2017-05-08T14:09:21+05:30 INFO No non-zero metrics in the last 30s 2017-05-08T14:09:37+05:30 INFO File is inactive: FilePath\FileName. Closing because close_inactive of 1m0s reached. 2017-05-08T14:09:42+05:30 INFO Harvester started for file: FilePath\FileName 2017-05-08T14:09:51+05:30 INFO Non-zero metrics in the last 30s: registrar.states.update=2 filebeat.harvester.started=1 publish.events=2 filebeat.harvester.closed=1 registrar.writes=2
There is only file in the directory "FilePath", this file is not updated (I can not see any new lines in the file, also the file modified timestamp has not changed). Where as publish.events=2 suggests that this file is updated with two new records (this is my understanding of filebeat logs, could be totally wrong).
Registry file contains only one entry, similar to following
This file contains 5087232 character. 5087128 is the end of second last line.
Last line in this file does not contains line end character (CR-LF).
My understanding is that once the file is marked as inactive and no new changes are made to the file, no new harvester should be started. Am I missing something in prospector configuration or my understanding needs an update? Or absence of line end character in the last line is messing with filebeat?