High number of log files leads to high filebeat RAM and CPU usage

I have a filebeat client watching a log folder that receives up to 120,000 log files per day. Early in the day, when there are only a few files in the folder, filebeat works fine. However, later in the day filebeat CPU and RAM usage is very high. I believe this is because it is trying to scan all 120,000 files constantly. I moved the scan frequency to 30 seconds, but it seems that did not help.

What can I do to process these log files without the rise in CPU and RAM once the log folder file count gets high?

Any recommendations will help.

Thanks

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.