Filebeat has big delay to send log to Kafka

Elastic Stack Beats
1

Hi,

We have log server running filebeat to send log data to Kafka. The log file is on NFS mounted filesystem. we do logrotate hourly, the size of the file is ~18G at the peak time. we have log time and the time received on Kafka. sometimes the delay is up to 5000 seconds.
I tried update the filebeat.yml, like set 'max_proc' to 5, increase the worker to '5', etc. it helps a little, but still see huge delay.
Would anyone can help me with it?

Thanks,
Lee

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.