Filebeat is taking too long to pick up new files

[shankar1 (Shankar)]

Problem : Initially the time taken between posting a new file and picking up by the filebeat harvester was fast (seconds). After a week I realized that filebeat speed of picking up got reduced (minutes / hours) and few files got ignored in this process because of "ignore_older". Please suggest what are the other options to be added in filebeat config. Currency registry size is only increasing.

Filebeat version : Filebeat (5.5.0)
Data Flow : Filebeat to Logstash. File needs to be read only once and there is no file rotation.

filebeat.prospectors:
-input_type: log
-D:\prod\*\*.log
-ignore_older: 48h
-clean_inactive: 70h
-scan_frequency: 20s
-close_eof: true

[steffens (Steffen Siering)]

Can you share your complete filebeat configuration?

Are files rotated?

How do you determine when filebeat is picking up files?

[shankar1 (Shankar)]

filebeat.prospectors:
-input_type: log
-D:\prod\date-*\*.log
-ignore_older: 48h
-clean_inactive: 70h
-scan_frequency: 20s
-close_eof: true
hosts: ["10.1**.1**.1**:5045","10.1**.1**.1**:5045","10.1**.1**.1**:5045"]
loadbalance: true

No File rotation

Filebeat registry file and printing log on CMD

[shankar1 (Shankar)]

When I restart "Flushing Spooler timeout" goes on for a long time.