Filebeat CPU and RAM utilization are all over the place

[Elay17]

I am currently running ECK 8.6.1 on a bare metal cluster, but I'm experiencing some issues with Filebeats. Depending on the configuration, it either leaks RAM or utilizes an excessive amount of CPU. Here is the configuration file I'm using:

apiVersion: beat.k8s.elastic.co/v1beta1
kind: Beat
metadata:
  name: filebeat
spec:
  type: filebeat
  version: 8.6.1
  elasticsearchRef:
    name: elasticsearch  
  config:
    filebeat:
      autodiscover:
        providers:
          - type: kubernetes
            add_resource_metadata:
              cronjob: false
              deployment: false
            node: ${NODE_NAME}
            templates:
              - condition.or:
                  - equals:
                      kubernetes.namespace: "namespace1"  
                  - equals:
                      kubernetes.namespace: "namespace2"                       
                config:
                - type: container
                  paths:
                    - /var/log/containers/*-${data.kubernetes.container.id}.log
                  index: "filebeat-${data.kubernetes.namespace}-%{+yyyy.MM.dd}"
                  exclude_files:
                  - ".*rabbitmq.*"        
                  - ".*portal.*"    
                  - ".*businesscard.*"        
                  - ".*airflow.*"
                  - ".*redis.*" 
                  - ".*mongo.*" 
                  - ".*frontend-app.*"                  
                  - ".*rook.*"
                  - ".*filbeat.*"
                  - ".*statistics.*"
                  processors:            
                  - decode_json_fields:
                      fields: ["message"]
                      process_array: false
                      max_depth: 3
                      target: ""
                      overwrite_keys: true
                      add_error_key: true
                      expand_keys: true
  daemonSet:
    podTemplate:
      spec:
        serviceAccountName: filebeat
        automountServiceAccountToken: true
        terminationGracePeriodSeconds: 30
        dnsPolicy: ClusterFirstWithHostNet
        hostNetwork: true # Allows to provide richer host metadata
        containers:
        - name: filebeat
          securityContext:
            runAsUser: 0
          resources:
            limits:
              memory: 800Mi
              cpu: 2000m         
          volumeMounts:
          - name: varlogcontainers
            mountPath: /var/log/containers
          - name: varlogpods
            mountPath: /var/log/pods
          - name: varlibdockercontainers
            mountPath: /var/lib/docker/containers
          env:
            - name: NODE_NAME
              valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName
        volumes:
        - name: varlogcontainers
          hostPath:
            path: /var/log/containers
        - name: varlogpods
          hostPath:
            path: /var/log/pods
        - name: varlibdockercontainers
          hostPath:
            path: /var/lib/docker/containers

With this configuration, CPU utilization goes through the roof, the metrics for the Filebeat daemonset are as follows:

When I remove 'add_resource_metadata.deployment: false' from the configuration, while keeping 'cronjob: false', the CPU utilization returns to normal. However, in this case, Filebeats begins to leak RAM and eventually gets OOMKilled within a few days.

As I am verynew to ECK, I may be overlooking something, and my configuration is definitely flawed. Any assistance in resolving this issue would be greatly appreciated!

[Elay17]

Here's how metrics looks like after I remove 'add_resource_metadata.deployment: false' and Filebeats started to leak memory

[germain_nganko]

We have a similar behaviour, with the exception that even when we have 'add_resource_metadata.deployment: false' in the configuration, while keeping 'cronjob: false', We still observe memory leakadge.