# Filebeat is using too much cpu **URL:** \ **Category:** Beats\ **Tags:** filebeat\ **Created:** [June 28, 2019, 6:33am UTC](https://discuss.elastic.co/t/filebeat-is-using-too-much-cpu/187978 "2019-06-28T06:33:14Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![biello](https://avatars.discourse-cdn.com/v4/letter/b/4da419/32.png) [@biello](https://discuss.elastic.co/u/biello)\ **Post date:** [June 28, 2019, 6:33am UTC](https://discuss.elastic.co/t/filebeat-is-using-too-much-cpu/187978/1 "2019-06-28T06:33:14Z")
![image](https://us1.discourse-cdn.com/elastic/original/3X/8/0/80d4af58c048f068743c9d8afd74db3d83c249db.jpeg) I didn't set the scan\_frequency option(which means leave as default 10s), why does it still use that much cpu? Here is the log from filebeat: ```auto 2019-06-28T14:30:52.034+0800 INFO [monitoring] log/log.go:144 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":47740,"time":{"ms":1782}},"total":{"ticks":800400,"time":{"ms":29664},"value":800400},"user":{"ticks":752660,"time":{"ms":27882}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":46},"info":{"ephemeral_id":"89e3b738-7016-49fa-af1c-3b7b560ea1b3","uptime":{"ms":810044}},"memstats":{"gc_next":67501680,"memory_alloc":46935904,"memory_total":150198960864}},"filebeat":{"events":{"active":107,"added":120138,"done":120031},"harvester":{"closed":1,"open_files":37,"running":37,"started":3}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"acked":120026,"batches":59,"total":120026}},"outputs":{"kafka":{"bytes_write":21317697}},"pipeline":{"clients":1,"events":{"active":1163,"filtered":5,"published":120184,"total":120188},"queue":{"acked":120026}}},"registrar":{"states":{"cleanup":1,"current":47,"update":120031},"writes":{"success":59,"total":59}},"system":{"load":{"1":10.6,"15":10.48,"5":10.4,"norm":{"1":0.6625,"15":0.655,"5":0.65}}}}}} ``` ---
**Author:** ![biello](https://avatars.discourse-cdn.com/v4/letter/b/4da419/32.png) [@biello](https://discuss.elastic.co/u/biello)\ **Post date:** [June 28, 2019, 7:00am UTC](https://discuss.elastic.co/t/filebeat-is-using-too-much-cpu/187978/2 "2019-06-28T07:00:42Z")
I'm using filebeat7.1 and here is the filebeat.yml, any idea? ```auto filebeat.inputs: - type: log paths: - /var/lib/docker/containers/*/*.log tail_files: true close_inactive: 5m close_timeout: 5m processors: #- add_cloud_metadata: ~ - add_locale: ~ - add_docker_metadata: host: "unix:///var/run/docker.sock" - add_host_metadata: netinfo.enabled: true cache.ttl: 5m - drop_event: when: or: - equals: container.labels.io_kubernetes_container_name: kube-system - equals: container.labels.io_kubernetes_container_name: default - equals: container.labels.io_kubernetes_container_name: kube-public - decode_json_fields: fields: ["message"] process_array: false max_depth: 1 overwrite_keys: false - include_fields: fields: ["host.hostname", "container", "message", "event"] #------------------------------- Kafka output ---------------------------------- output.kafka: versions: 2.1.0 hosts: ${KAFKA_HOSTS:?environment variable KAFKA_HOSTS not found} topic: ${KAFKA_TOPIC:?environment variable KAFKA_TOPIC not found} worker: 1 keep-alive: 600 required_acks: 0 compression: gzip max_message_bytes: 52428800 channel_buffer_size: 8192 bulk_max_size: 2048 compression_level: 4 ``` ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [June 29, 2019, 6:55am UTC](https://discuss.elastic.co/t/filebeat-is-using-too-much-cpu/187978/3 "2019-06-29T06:55:42Z")
Filebeat was designed to use as little resources as possible and be very light weight. To get there a lot of the processing that Logstash could do was not available and this had to be performed elsewhere, e.g. in Logstash or through an ingest pipeline. Now Filebeat is getting additional capabilities e.g. adding metadata and parsing JSON payload, and this all requires CPU cycles. When all of these are used it will therefore use more resources than older, less capable versions would. You seem to be doing a bit of processing. How large are your events? How many events per second does Filebeat process when using that amount of CPU? ---
**Author:** ![biello](https://avatars.discourse-cdn.com/v4/letter/b/4da419/32.png) [@biello](https://discuss.elastic.co/u/biello)\ **Post date:** [July 8, 2019, 6:23am UTC](https://discuss.elastic.co/t/filebeat-is-using-too-much-cpu/187978/4 "2019-07-08T06:23:29Z")
Got it, i have 10k events per second with event size about 500B on average, the cpu usage sounds reasonable now, I will try to remove the json decode processor. I understand parsing JSON payload requires CPU cycles, but why is adding metadata a cpu costing job? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [July 8, 2019, 7:13am UTC](https://discuss.elastic.co/t/filebeat-is-using-too-much-cpu/187978/5 "2019-07-08T07:13:49Z")
That I can not tell as I am not familiar with how metadata is added and cached. ---
**Author:** ![fredrik.jonsson](https://avatars.discourse-cdn.com/v4/letter/f/c2a13f/32.png) [@fredrik.jonsson](https://discuss.elastic.co/u/fredrik.jonsson)\ **Post date:** [July 8, 2019, 10:20am UTC](https://discuss.elastic.co/t/filebeat-is-using-too-much-cpu/187978/6 "2019-07-08T10:20:54Z")
This seems to be related to this issue, > [@CPU usage 100% when Elasticsearch is down](https://discuss.elastic.co/t/cpu-usage-100-when-elasticsearch-is-down/185850): > > I noticed that when Elasticsearch can't retrieve any more data the beats sending data went up to 100% CPU usage. E.g. if the disc is full on the Elasticsearch server. After clear up some space and "reset" Elasticsearch to be able to retrive data the CPU for the beats went down to normal behaviour. It is never OK to use that much CPU and risk the other process running on the same machine. Better loosing data than that the services stop working! Is this a bug or can I config the beats in a be… ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [August 5, 2019, 10:21am UTC](https://discuss.elastic.co/t/filebeat-is-using-too-much-cpu/187978/7 "2019-08-05T10:21:04Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.