# Filebeat consumes a large amount of disk io reads on a Kubernetes node **URL:** \ **Category:** Beats\ **Tags:** docker, filebeat\ **Created:** [May 7, 2019, 3:49pm UTC](https://discuss.elastic.co/t/filebeat-consumes-a-large-amount-of-disk-io-reads-on-a-kubernetes-node/180022 "2019-05-07T15:49:46Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![Max\_Brain](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/max_brain/32/45665_2.png) [@Max\_Brain](https://discuss.elastic.co/u/Max_Brain)\ **Post date:** [May 7, 2019, 3:49pm UTC](https://discuss.elastic.co/t/filebeat-consumes-a-large-amount-of-disk-io-reads-on-a-kubernetes-node/180022/1 "2019-05-07T15:49:46Z")
When running Filebeat on a Kubernetes node, the system is spending a large amount of cpu cycles on iowait. According to according to iotop, the system during this time is writing 10 MB/S. Reads account for upwards of 150 MB/S. The Filebeat container gradually uses more memory in conjunction with increasing the io load until the memory limit we setup on the pod is reached and the pod gets restarted. There are no logs in the Filebeat container that indicate that the pod is getting into an error state or is failing to send logs to Kafka. Is there a setting we are overlooking that is causing this? Here is what we have for the two input types used by the Filebeat container. The node is using the Docker json logging driver and is using the default file size and count. ``` - type: docker combine_partial: true cri.parse_flags: true close_inactive: 48h containers.ids: - "*" exclude_lines: - 1 . . - 8 - type: log paths: - "path 1" - "path 2" - "path 3" exclude_lines: - 1 . . - 5 fields: log_topic: 'log_topic_name' fields_under_root: true scan_frequency: 1s ``` ---
**Author:** ![Michal\_Pristas](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/michal_pristas/32/46639_2.png) [@Michal\_Pristas](https://discuss.elastic.co/u/Michal_Pristas)\ **Post date:** [May 9, 2019, 1:29pm UTC](https://discuss.elastic.co/t/filebeat-consumes-a-large-amount-of-disk-io-reads-on-a-kubernetes-node/180022/2 "2019-05-09T13:29:39Z")
can you please use `` button to correctly format the configuration? ---
**Author:** ![Max\_Brain](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/max_brain/32/45665_2.png) [@Max\_Brain](https://discuss.elastic.co/u/Max_Brain)\ **Post date:** [May 9, 2019, 4:21pm UTC](https://discuss.elastic.co/t/filebeat-consumes-a-large-amount-of-disk-io-reads-on-a-kubernetes-node/180022/3 "2019-05-09T16:21:14Z")
Formatting has been corrected. ---
**Author:** ![Michal\_Pristas](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/michal_pristas/32/46639_2.png) [@Michal\_Pristas](https://discuss.elastic.co/u/Michal_Pristas)\ **Post date:** [May 10, 2019, 12:55pm UTC](https://discuss.elastic.co/t/filebeat-consumes-a-large-amount-of-disk-io-reads-on-a-kubernetes-node/180022/4 "2019-05-10T12:55:20Z")
is the docker spec under k8s template? if so you may want to avoid having `*` for containers.id as for note in the docs: See warning above this section [https://www.elastic.co/guide/en/beats/filebeat/7.0/configuration-autodiscover.html#\_kubernetes](https://www.elastic.co/guide/en/beats/filebeat/7.0/configuration-autodiscover.html#_kubernetes) ---
**Author:** ![Max\_Brain](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/max_brain/32/45665_2.png) [@Max\_Brain](https://discuss.elastic.co/u/Max_Brain)\ **Post date:** [May 10, 2019, 5:11pm UTC](https://discuss.elastic.co/t/filebeat-consumes-a-large-amount-of-disk-io-reads-on-a-kubernetes-node/180022/5 "2019-05-10T17:11:07Z")
The spec is part of a Kubernetes resource definition that Filebeat reads from. We do not have use autodiscover. Should we use it instead of looking for all container ids that match `*` ? According to the Docker input documentation, it tells us to use `containers.ids: '*'` to read from all containers. ---
**Author:** ![Michal\_Pristas](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/michal_pristas/32/46639_2.png) [@Michal\_Pristas](https://discuss.elastic.co/u/Michal_Pristas)\ **Post date:** [May 21, 2019, 7:25am UTC](https://discuss.elastic.co/t/filebeat-consumes-a-large-amount-of-disk-io-reads-on-a-kubernetes-node/180022/6 "2019-05-21T07:25:12Z")
sorry I made wrong assumption. also filebeat use to consume more memory and cpu cycles with more and more files harvested. do you know how many files are being processed ? ---
**Author:** ![Max\_Brain](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/max_brain/32/45665_2.png) [@Max\_Brain](https://discuss.elastic.co/u/Max_Brain)\ **Post date:** [May 23, 2019, 6:11pm UTC](https://discuss.elastic.co/t/filebeat-consumes-a-large-amount-of-disk-io-reads-on-a-kubernetes-node/180022/7 "2019-05-23T18:11:42Z")
We are creating roughly 5-8 new files every minute due to docker logs getting rotated. We have at most 42 docker log files. We have 10-12 Harvesters running when Filebeat starts consuming a lot of IO. Filebeat has around 900 open files, of which are docker log files that have been deleted, but Filebeat still holds open the inode. Filebeat does not show any error messages. It only shows metrics being published. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [June 20, 2019, 6:11pm UTC](https://discuss.elastic.co/t/filebeat-consumes-a-large-amount-of-disk-io-reads-on-a-kubernetes-node/180022/8 "2019-06-20T18:11:51Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.