I have a filebeat client watching a log folder that receives up to 120,000 log files per day. Early in the day, when there are only a few files in the folder, filebeat works fine. However, later in the day filebeat CPU and RAM usage is very high. I believe this is because it is trying to scan all 120,000 files constantly. I moved the scan frequency to 30 seconds, but it seems that did not help.
What can I do to process these log files without the rise in CPU and RAM once the log folder file count gets high?
Any recommendations will help.
Thanks