I am having a Status-Logfile, that changes periodically (every minute). The problem is, that mostly only content of the file changes (timestamps, some numbers) and the size of the file stays the same. I am having trouble configuring the filebeat to read this file, as the harvesting is mostly triggered by the file size changes.
I was trying settings like: scan_frequency, close_eof, tail_on_rotate , but no success.
Would it be possible to configure filebeat to act based on file modification time, rather then file size? If yes, how?
It is not exactly about size, the thing is that Filebeat collects lines appended to files, that is the usual behaviour of log files. In your case, if the lines are replacing the current content, instead of being appended, they are going to be ignored by Filebeat.
One thing you can try is to configure your input with the clean_inactive option. With this option you can make filebeat to "forget" about already read files. Once "forgotten", filebeat will read them again if they are rewritten. Be careful with this option because it needs to be configured along some some other settings to work correctly.
Other option, if you are the author of the application that creates this log file, is to consider a different rotation strategy for these logs. For example Filebeat is able to correctly handle log files that are moved or renamed before being written again.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.
