Metricbeat in Kubernetes only get processes from the container and not the host (Windows)

Elastic Stack Beats
1

Context
I am trying to setup metricbeat 7.12 as a DaemonSet in an Azure Kubernetes Cluster on a Nano Windows Server host. I based my Kubernetes config of metricbeat on this document.

In this document, the ConfigMap named "metricbeat-daemonset-modules" contains the configuration for the module system. The metricset "process" is enabled (among others).

Expected behavior
From what I understood with this config, I would have expected metricbeat to look at all the processes on the Host and give me the ones with highest CPU/memory.

Observed behavior
What I see instead is that metricbeat only seems to look at the processes on the container where metricbeat is currently running on.

Question
What is the expected behavior? Is there a way for metricbeat to look at all the process from the Host perspective?

Details
Please note that:

  • Metribeat pods are running in an "elasticsearch" namespace and not in "kube-system"
  • I have metricbeat configured as a DaemonSet for a Linux host and another one for Windows Nano server host. The problem seems only related to Windows host.

Here is my system module's config:

    apiVersion: v1
    kind: ConfigMap
    metadata:
      name: metricbeat-daemonset-modules
      labels:
        k8s-app: metricbeat
    data:
      system.yml: |-
        - module: system
          period: 10s
          metricsets:
            - cpu
            - load
            - memory
            - network
            - process
            - process_summary
            #- core
            #- diskio
            #- socket
          processes: ['*']
          process.include_top_n:
            by_cpu: 20      # include top 5 processes by CPU
            by_memory: 20   # include top 5 processes by memory

        - module: system
          period: 1m
          metricsets:
            - filesystem
            - fsstat
          processors:
          - drop_event.when.regexp:
              system.filesystem.mount_point: '^/(sys|cgroup|proc|dev|etc|host|lib)($|/)'
      kubernetes.yml: |-
        - module: kubernetes
          metricsets:
            - node
            - system
            - pod
            - container
            - volume
          period: 10s
          host: ${NODE_NAME}
          hosts: ["https://${HOSTNAME}:10250"]
          bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
          ssl.certificate_authorities:
            - /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
          ssl.verification_mode: "none"
        - module: kubernetes
          metricsets:
            - proxy
          period: 10s
          host: ${NODE_NAME}
          hosts: ["localhost:10249"]
3

Hi!

I think the -system.hostfs=/hostfs at beats/metricbeat-kubernetes.yaml at a0d3c74c445187936802c5f87d3bdc19e6955e04 · elastic/beats · GitHub should do the trick. Do you start metricbeat like this?
Also make sure that you mount the underlying fs into the container like at beats/metricbeat-kubernetes.yaml at master · elastic/beats · GitHub.

Find more info at Run Metricbeat on Docker | Metricbeat Reference [7.12] | Elastic

4

Hey Chris,
Thanks for the reply. The yaml you are referring to seems to be only working for a DaemonSet deployed on a Linux host. In my case, it is deployed on a Windows host as mentioned in my previous post.

I understand now that metricbeat gets the processes info from /proc on a Linux host. I guess that won't work on a Windows host.

Do you know if metricbeat running on a Windows Host in a Kubernetes cluster was ever tested? Should there be a mention maybe in the documentation about certain features not supported on Windows Host?

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.