Filebeat consumes a large amount of disk io reads on a Kubernetes node

[Max_Brain]

When running Filebeat on a Kubernetes node, the system is spending a large amount of cpu cycles on iowait. According to according to iotop, the system during this time is writing 10 MB/S. Reads account for upwards of 150 MB/S. The Filebeat container gradually uses more memory in conjunction with increasing the io load until the memory limit we setup on the pod is reached and the pod gets restarted. There are no logs in the Filebeat container that indicate that the pod is getting into an error state or is failing to send logs to Kafka. Is there a setting we are overlooking that is causing this? Here is what we have for the two input types used by the Filebeat container. The node is using the Docker json logging driver and is using the default file size and count.

- type: docker
        combine_partial: true
        cri.parse_flags: true
        close_inactive: 48h
        containers.ids:
        - "*"
        exclude_lines:
        - 1
        .
        .
        - 8
- type: log
        paths:
        - "path 1"
        - "path 2"
        - "path 3"
        exclude_lines:
        - 1
        .
        .
        - 5
        fields:
          log_topic: 'log_topic_name'
        fields_under_root: true
        scan_frequency: 1s

[Michal_Pristas]

can you please use </> button to correctly format the configuration?

[Max_Brain]

Formatting has been corrected.

[Michal_Pristas]

is the docker spec under k8s template? if so you may want to avoid having * for containers.id as for note in the docs:
See warning above this section https://www.elastic.co/guide/en/beats/filebeat/7.0/configuration-autodiscover.html#_kubernetes

[Max_Brain]

The spec is part of a Kubernetes resource definition that Filebeat reads from. We do not have use autodiscover. Should we use it instead of looking for all container ids that match * ? According to the Docker input documentation, it tells us to use containers.ids: '*' to read from all containers.

[Michal_Pristas]

sorry I made wrong assumption. also filebeat use to consume more memory and cpu cycles with more and more files harvested. do you know how many files are being processed ?

[Max_Brain]

We are creating roughly 5-8 new files every minute due to docker logs getting rotated. We have at most 42 docker log files. We have 10-12 Harvesters running when Filebeat starts consuming a lot of IO. Filebeat has around 900 open files, of which are docker log files that have been deleted, but Filebeat still holds open the inode. Filebeat does not show any error messages. It only shows metrics being published.