I an new to Elastic world and have been reading the documents and analyzing the current ELK setup. Filebeat version is 6.0.
Question # 1: Harvestor is created for each file. Harvestor is closed if it is inactive. The close_inactive in current setup has default value of 5m0s. clean_removed, clean_inactive,ignore_older, close_renamed, close_removed, close_timeout have default values. As per the document - How Filebeat works | Filebeat Reference [6.0] | Elastic, harvesting of file (if the hearvestor was closed) would start after scan_frequency is met. Does that mean, if a harvestor is closed, then after scan_frequency is met it will open a harvestor again. I checked the log files and didnt find such behavior. Which means, Filebeat knows not to open the file again because it has been processed. How?
Question #2 - Frequently asked questions | Filebeat Reference [6.0] | Elastic, mentions what needs to be done when registry is too big. How to define if registry is too big, is 100kb registry file big or is it 1mb or some other number?
Question #3 - I started filebeat on a server that has existing log files. ignore_older option was not set and hence harvestor is getting created for older files too. Is there a method in selecting files for harvestor or filebeat will randomly pick files to create harvestor
Please let me know if additional information is required.