# Filebeat consuming too much CPU **URL:** \ **Category:** Beats\ **Tags:** docker, filebeat\ **Created:** [December 18, 2022, 4:33pm UTC](https://discuss.elastic.co/t/filebeat-consuming-too-much-cpu/321497 "2022-12-18T16:33:08Z")\ **Posts on this page:** 17\ **Page:** 1
**Author:** ![Priya\_Vardhan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/priya_vardhan/32/99176_2.png) [@Priya\_Vardhan](https://discuss.elastic.co/u/Priya_Vardhan)\ **Post date:** [December 18, 2022, 4:33pm UTC](https://discuss.elastic.co/t/filebeat-consuming-too-much-cpu/321497/1 "2022-12-18T16:33:08Z")
Hi, Installed filebeat using docker in 3 servers. In each of the servers, CPU reaching to 99% the moment filebeat starts harvesting. Please suggest how we can control CPU usage ---
**Author:** ![Priya\_Vardhan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/priya_vardhan/32/99176_2.png) [@Priya\_Vardhan](https://discuss.elastic.co/u/Priya_Vardhan)\ **Post date:** [December 19, 2022, 3:14pm UTC](https://discuss.elastic.co/t/filebeat-consuming-too-much-cpu/321497/2 "2022-12-19T15:14:37Z")
Can you please help on this? ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [December 19, 2022, 3:35pm UTC](https://discuss.elastic.co/t/filebeat-consuming-too-much-cpu/321497/3 "2022-12-19T15:35:45Z")
You need to provide more context. Please share your `filebeat.yml` for every server. ---
**Author:** ![Priya\_Vardhan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/priya_vardhan/32/99176_2.png) [@Priya\_Vardhan](https://discuss.elastic.co/u/Priya_Vardhan)\ **Post date:** [December 19, 2022, 6:28pm UTC](https://discuss.elastic.co/t/filebeat-consuming-too-much-cpu/321497/4 "2022-12-19T18:28:12Z")
Hi @leandrojmp , Please find the filebeat.yml below: filebeat.inputs: -type: filestream Id: Test log Paths: "/var/lib/test.log" "/usr/share/test.log" scan\_frequency: 5m close\_inactive: 4m ignore\_older: 7d There are so many log paths for which created so many input filestream ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [December 19, 2022, 6:55pm UTC](https://discuss.elastic.co/t/filebeat-consuming-too-much-cpu/321497/5 "2022-12-19T18:55:13Z")
> [@Priya\_Vardhan](#): > > There are so many log paths for which created so many input filestream How many? What are the specs of the filebeat server? This may be the issue. ---
**Author:** ![Priya\_Vardhan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/priya_vardhan/32/99176_2.png) [@Priya\_Vardhan](https://discuss.elastic.co/u/Priya_Vardhan)\ **Post date:** [December 19, 2022, 7:12pm UTC](https://discuss.elastic.co/t/filebeat-consuming-too-much-cpu/321497/6 "2022-12-19T19:12:25Z")
Suppose there are 15 filestream types given, under each there will be around 10 log paths mentioned. The CPU is 2 cores in filebeat monitoring server, having memory of 8GB @leandrojmp Any suggestions please? ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [December 19, 2022, 8:53pm UTC](https://discuss.elastic.co/t/filebeat-consuming-too-much-cpu/321497/7 "2022-12-19T20:53:24Z")
> [@Priya\_Vardhan](#): > > filebeat.inputs: > > -type: filestream > Id: Test log > Paths: > "/var/lib/test.log" > "/usr/share/test.log" > scan\_frequency: 5m > close\_inactive: 4m > ignore\_older: 7d So clearly according to this > [@Priya\_Vardhan](#): > > Suppose there are 15 filestream types given, under each there will be around 10 log paths mentioned. That above is not your full filebeat.yml So are you saying that you have ~150 Log Paths? 15 Inputs x 10 Paths? ---
**Author:** ![Priya\_Vardhan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/priya_vardhan/32/99176_2.png) [@Priya\_Vardhan](https://discuss.elastic.co/u/Priya_Vardhan)\ **Post date:** [December 19, 2022, 9:04pm UTC](https://discuss.elastic.co/t/filebeat-consuming-too-much-cpu/321497/8 "2022-12-19T21:04:56Z")
Hi @stephenb , yes, I have around 150 log paths ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [December 19, 2022, 9:16pm UTC](https://discuss.elastic.co/t/filebeat-consuming-too-much-cpu/321497/9 "2022-12-19T21:16:01Z")
When filebeat starts up it will try to read every log file... even the whole history... is that your desired requirement? How much actual log file data are there? in term of GBs And you said ~150 Logs paths are there more than 1 file per log path? do you use any `*` in the paths or files? Do you want to ignore older files? Or Load them All? What version of Filebeat and Elasticsearch? What size resources (CPU, RAM etc) for filebeat? ---
**Author:** ![Priya\_Vardhan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/priya_vardhan/32/99176_2.png) [@Priya\_Vardhan](https://discuss.elastic.co/u/Priya_Vardhan)\ **Post date:** [December 19, 2022, 9:26pm UTC](https://discuss.elastic.co/t/filebeat-consuming-too-much-cpu/321497/10 "2022-12-19T21:26:57Z")
Yes, it has to read all the log files one after the other but the whole history is not required. It has to read only new logs every time. But if I have to look at history before 7 days, I should be able to get those logs. There will be around 50GB log data. I used the actual path of file and only for 20+ paths, used \* in the paths I want to load new logs everytime , but if I check the history before 7days, I should be able to get that. Filebeat version: 8.1.3 Elastic search: 8.0.0 CPU is 2 cores in filebeat monitoring server, around 8GB memory ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [December 19, 2022, 9:41pm UTC](https://discuss.elastic.co/t/filebeat-consuming-too-much-cpu/321497/11 "2022-12-19T21:41:16Z")
> [@Priya\_Vardhan](#): > > It has to read only new logs every time. But if I have to look at history before 7 days, I should be able to get those logs. you can use [ignore\_older](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-filestream.html#filebeat-input-filestream-ignore-older) setting to skip the older files.. that can be set per input. Then if you need to go back further you should be able to unset it on the input you want. But yes is you start up filebeat with many "prospectors / harvestors" which is what you are doing it will consume CPU and RAM ... until it catches up that 50GB ... How much GB / day once you catch up? ---
**Author:** ![Priya\_Vardhan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/priya_vardhan/32/99176_2.png) [@Priya\_Vardhan](https://discuss.elastic.co/u/Priya_Vardhan)\ **Post date:** [December 19, 2022, 10:26pm UTC](https://discuss.elastic.co/t/filebeat-consuming-too-much-cpu/321497/12 "2022-12-19T22:26:02Z")
I used ignore\_older and set it to 7d. I didn't check the exact GB it is consuming but when filebeat container is up, within next few minutes CPU will be 99% ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [December 19, 2022, 10:54pm UTC](https://discuss.elastic.co/t/filebeat-consuming-too-much-cpu/321497/13 "2022-12-19T22:54:07Z")
> [@stephenb](#): > > How much GB / day once you catch up? I mean how many GBs / day of logs... once the logs catch up... I suspect filebeat is just trying to catch up.... with the _many_ logs paths / prospectors and volume... ---
**Author:** ![Priya\_Vardhan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/priya_vardhan/32/99176_2.png) [@Priya\_Vardhan](https://discuss.elastic.co/u/Priya_Vardhan)\ **Post date:** [December 19, 2022, 11:55pm UTC](https://discuss.elastic.co/t/filebeat-consuming-too-much-cpu/321497/14 "2022-12-19T23:55:46Z")
It is taking around 17GB per day ---
**Author:** ![hendry.lim](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hendry.lim/32/71328_2.png) [@hendry.lim](https://discuss.elastic.co/u/hendry.lim)\ **Post date:** [December 20, 2022, 1:46am UTC](https://discuss.elastic.co/t/filebeat-consuming-too-much-cpu/321497/15 "2022-12-20T01:46:59Z")
Since you only have 2 cores in the host that Filebeat is running, you may want to try to limit it to only one core using the `max_procs` configuration in `filebeat.yml` [Configure general settings | Filebeat Reference [master] | Elastic](https://www.elastic.co/guide/en/beats/filebeat/master/configuration-general-options.html#_max_procs). However, you may want to enable monitoring on Filebeat to ensure that Filebeat is able to catch up with the amount of logs you aim to ingest. ---
**Author:** ![Priya\_Vardhan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/priya_vardhan/32/99176_2.png) [@Priya\_Vardhan](https://discuss.elastic.co/u/Priya_Vardhan)\ **Post date:** [December 20, 2022, 8:30am UTC](https://discuss.elastic.co/t/filebeat-consuming-too-much-cpu/321497/16 "2022-12-20T08:30:20Z")
Hi @hendry.lim , Okay sure, I will try using the max\_process and check how CPU is behaving. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [January 17, 2023, 10:31am UTC](https://discuss.elastic.co/t/filebeat-consuming-too-much-cpu/321497/17 "2023-01-17T10:31:10Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.