We have log server running filebeat to send log data to Kafka. The log file is on NFS mounted filesystem. we do logrotate hourly, the size of the file is ~18G at the peak time. we have log time and the time received on Kafka. sometimes the delay is up to 5000 seconds.
I tried update the filebeat.yml, like set 'max_proc' to 5, increase the worker to '5', etc. it helps a little, but still see huge delay.
Would anyone can help me with it?