What happens to the old files of the previous day?
For the old files, it seems that ignore_older will become tricky. One thing you could try for the old files it using the -once param and run filebeat only once until completion?
I have done several tests and I prefer to merge all that stuff in this topic.
On a test server, I have indexed all the files "old" using the -once option and the following config (the ignore_older property is set to only index files after a given date):
All the source directories are Windows mounted drives on shares directories from two distinct servers.
After indexing the "old" files, I started filebeat without the -once option and with quite the same config as the previous one except on this :
ignore_older: 4080h
replaced by this :
ignore_older: 10m
clean_inactive: 15m
The registry file size is now 40MB.
I do not encounter the "very long prospector loading" issue when restarting filebeat as described in the other topic.
But I have sometimes a quite big delay between the time a new file is copied to a watched directory and the time it is received by logstash (which is installed on the same server as filebeat for test purpose). The delay can be up to 15 minutes.
The files with biggest delays are not multiline. There is only one csv line inside, and sometimes a few lines.
I don't see nothing special in the log files. I think the main problem is the amount of files in the directories that filebeat has to monitor.
As filebeat is monitoring files on two shared directories, I wanted to change the architecture and install filebeat directly on the servers that have the shared directories. But these servers are NAS and I can't install filebeat on them.
I wonder if this could be a good idea to have multiple filebeat instances that monitor a few directories instead of one instance that monitor many directories. This could decrease the registry size (which is almost 50MB actually). I will test it.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.