Metricbeat Elasticsearch module unexpected memory usage on master

[Charles_Ferguson]

Hello, I am having an issue with trying to configure metricbeats Elasticsearch module. The specific issue I am seeing is that I have configured a daemonset of metricbeats to collect node and nodes_stats metrics from all of my nodes but the one that is on the same node as my master has about at least a magnitude higher memory usage and frequently is stopped to out of memory errors. From this daemonset I am only collecting the node and node_stats metricsets which I believe should be for the local node only.

I use the cloud-on-k8s operator/beat CRD to deploy my daemonset and it looks like below.

---
apiVersion: beat.k8s.elastic.co/v1beta1
kind: Beat
metadata:
  name: metricbeat
spec:
  type: metricbeat
  version: 7.9.1
  elasticsearchRef:
    name: monitoring
    namespace: monitoring
  kibanaRef:
    name: monitoring
    namespace: monitoring
  config:
    setup:
      ilm:
        enabled: auto
        rollover_alias: "metricbeat-7.9.1"
        pattern: "{now/d}-000001"
    logging.json: true
    metricbeat:
      autodiscover:
        providers:
        - type: kubernetes
          scope: node
          hints.enabled: true
          templates:
          - condition:
              equals:
                kubernetes.labels.app: elasticsearch
            config:
            - hosts: https://${data.host}:9200
              metricsets:
              - node
              - node_stats
              module: elasticsearch
              password: ${MONITORED_ES_PASSWORD}
              period: 10s
              ssl.certificate_authorities:
              - /mnt/monitored-elastic/ca.crt
              ssl.verification_mode: none
              username: ${MONITORED_ES_USERNAME}
              xpack.enabled: true
              scope: node
          - condition:
              equals:
                kubernetes.labels.app: kibana
            config:
            - hosts: http://${data.host}:5601
              metricsets:
              - stats
              module: kibana
              password: ${MONITORED_ES_PASSWORD}
              period: 10s
              ssl.certificate_authorities:
              - /mnt/monitored-elastic/ca.crt
              ssl.verification_mode: none
              username: ${MONITORED_ES_USERNAME}
              xpack.enabled: true
          - condition:
              equals:
                kubernetes.labels.app: logstash
            config:
            - hosts: http://${data.host}:9600
              metricsets:
              - node
              - node_stats
              module: logstash
              password: ${MONITORED_ES_PASSWORD}
              period: 10s
              ssl.verification_mode: none
              username: ${MONITORED_ES_USERNAME}
              xpack.enabled: true
      modules:
      - module: kubernetes
        period: 30s
        host: ${NODE_NAME}
        hosts:
        - https://${NODE_NAME}:10250
        bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
        ssl:
          verification_mode: none
        metricsets:
        - node
        - system
        - pod
        - container
        - volume
      - module: system
        period: 30s
        metricsets:
        - cpu
        - load
        - memory
        - network
        - process
        - process_summary
        process:
          include_top_n:
            by_cpu: 5
            by_memory: 5
        processes:
        - .*
      - module: system
        period: 1m
        metricsets:
        - filesystem
        - fsstat
        processors:
        - drop_event:
            when:
              regexp:
                system:
                  filesystem:
                    mount_point: ^/(sys|cgroup|proc|dev|etc|host|lib)($|/)
    processors:
    - add_cloud_metadata: {}
    - add_host_metadata: {}
  daemonSet:
    podTemplate:
      metadata: {}
      spec:
        automountServiceAccountToken: true
        containers:
        - name: metricbeat
          args:
          - -e
          - -c
          - /etc/beat.yml
          - -system.hostfs=/hostfs
          env:
          - name: NODE_NAME
            valueFrom:
              fieldRef:
                fieldPath: spec.nodeName
          envFrom:
          - secretRef:
              name: monitoring-metricbeat-environment
          resources:
            limits:
              memory: 256Mi
          volumeMounts:
          - mountPath: /mnt/monitored-elastic/
            name: monitored-elastic-certs
            readOnly: true
          - mountPath: /hostfs/sys/fs/cgroup
            name: cgroup
          - mountPath: /var/run/docker.sock
            name: dockersock
          - mountPath: /hostfs/proc
            name: proc
        securityContext:
          runAsUser: 0
        serviceAccountName: metricbeat
        terminationGracePeriodSeconds: 30
        volumes:
        - name: monitored-elastic-certs
          secret:
            secretName: monitored-es-http-certs-public
        - hostPath:
            path: /sys/fs/cgroup
          name: cgroup
        - hostPath:
            path: /var/run/docker.sock
          name: dockersock
        - hostPath:
            path: /proc
          name: proc

Ideally I would have a seperate metricbeat deployment collecting cluster wide metricsets directly from the master rather than having to add memory limits to the entire daemonset.