High memory usage of Metricbeat in Kubernetes

[beatman]

Hi,

Continuing the discussion from High memory usage of Metricbeat in Kubernetes:

I'm facing exactly the same problem with metricbeat consuming memory until it gets OOM-killed.
This happens on all three k8s clusters I'm running metricbeat on. I'm using version 7.17.
I also tried to run a dedicated metricbeat deployment for kube-state-metrics but it has no effect.
Actually, the KSM metricbeat deployment gets OOM-killed even faster than the other ones.
It usually takes a couple of hours then metricbeat gets OOM-killed..

I've set the resources.limits.memory to 5Gi, which is extremely high in my opinion.

Here is the configuration of the metricbeat DaemonSet as used in my self-created helm chart:

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: metricbeat-daemonset-config
  labels:
    {{- include "k8s-beats-monitoring.labels" . | nindent 4 }}
data:
  metricbeat.yml: |-
    monitoring:
      enabled: true
      cluster_uuid: "{{ .Values.elasticsearch.uuid }}"
      elasticsearch:
        hosts: {{ .Values.elasticsearch.hosts | toStrings }}
        username: ${ELASTICSEARCH_USERNAME}
        password: ${ELASTICSEARCH_PASSWORD}
        ssl.certificate_authorities: ["{{ .Values.elasticsearch.certPath }}/ca.crt"]
    metricbeat.config.modules:
      # Mounted `metricbeat-daemonset-modules` configmap:
      path: ${path.config}/modules.d/*.yml
      # Reload module configs as they change:
      reload.enabled: false

    metricbeat.autodiscover:
      providers:
        # To enable hints based autodiscover uncomment this:
        - type: kubernetes
          host: ${NODE_NAME}
          hints.enabled: true

    processors:
      - add_fields:
         target: kubernetes
         fields:
            cluster:
               name: "{{ .Values.kubernetes.cluster.name }}"
      - add_kubernetes_metadata:
      - add_host_metadata:
         cache.ttl: 5m
         geo:
           name: {{ .Values.kubernetes.cluster.name }}
      
    logging.level: warning

    cloud.id: ${ELASTIC_CLOUD_ID}
    cloud.auth: ${ELASTIC_CLOUD_AUTH}

    output.elasticsearch:
      hosts: {{ .Values.elasticsearch.hosts | toStrings }}
      loadbalance: true
      worker: 1
      username: ${ELASTICSEARCH_USERNAME}
      password: ${ELASTICSEARCH_PASSWORD}
      ssl:
         certificate_authorities: ["{{ .Values.elasticsearch.certPath }}/ca.crt"]
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: metricbeat-daemonset-modules
  labels:
    {{- include "k8s-beats-monitoring.labels" . | nindent 4 }}
data:
  system.yml: |-
    - module: system
      period: 10s
      metricsets:
        - cpu
        - load
        - memory
        - network
        - process
        - process_summary
        #- core
        #- diskio
        #- socket
      processes: ['.*']
      process.include_top_n:
        by_cpu: 5      # include top 5 processes by CPU
        by_memory: 5   # include top 5 processes by memory

    - module: system
      period: 1m
      metricsets:
        - filesystem
        - fsstat
      processors:
      - drop_event.when.regexp:
          system.filesystem.mount_point: '^/(sys|cgroup|proc|dev|etc|host|lib|snap)($|/)'
  kubernetes.yml: |-
    - module: kubernetes
      metricsets:
        - node
        - system
        - pod
        - container
        - volume
      period: 10s
      host: ${NODE_NAME}
      hosts: ["https://${NODE_NAME}:10250"]
      bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
      ssl.verification_mode: "none"
      add_metadata: true
    - module: kubernetes
      metricsets:
        - proxy
      period: 10s
      host: ${NODE_NAME}
      hosts: ["localhost:10249"]