Problem : Initially the time taken between posting a new file and picking up by the filebeat harvester was fast (seconds). After a week I realized that filebeat speed of picking up got reduced (minutes / hours) and few files got ignored in this process because of "ignore_older". Please suggest what are the other options to be added in filebeat config. Currency registry size is only increasing.
Filebeat version : Filebeat (5.5.0)
Data Flow : Filebeat to Logstash. File needs to be read only once and there is no file rotation.
filebeat.prospectors: -input_type: log -D:\prod\*\*.log -ignore_older: 48h -clean_inactive: 70h -scan_frequency: 20s -close_eof: true