Filebeat is taking too long to pick up new files


(Shankar) #1

Problem : Initially the time taken between posting a new file and picking up by the filebeat harvester was fast (seconds). After a week I realized that filebeat speed of picking up got reduced (minutes / hours) and few files got ignored in this process because of "ignore_older". Please suggest what are the other options to be added in filebeat config. Currency registry size is only increasing.

Filebeat version : Filebeat (5.5.0)
Data Flow : Filebeat to Logstash. File needs to be read only once and there is no file rotation.

filebeat.prospectors:
-input_type: log
-D:\prod\*\*.log
-ignore_older: 48h
-clean_inactive: 70h
-scan_frequency: 20s
-close_eof: true

(Steffen Siering) #2

Can you share your complete filebeat configuration?

Are files rotated?

How do you determine when filebeat is picking up files?


(Shankar) #3

filebeat.prospectors:
-input_type: log
-D:\prod\date-*\*.log
-ignore_older: 48h
-clean_inactive: 70h
-scan_frequency: 20s
-close_eof: true
hosts: ["10.1**.1**.1**:5045","10.1**.1**.1**:5045","10.1**.1**.1**:5045"]
loadbalance: true

No File rotation

Filebeat registry file and printing log on CMD


(Shankar) #4

When I restart "Flushing Spooler timeout" goes on for a long time.


(system) #5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.