How to monitor kubernetes node filesystem from a container

[Nicolas_Labrot]

Hi,

I'm trying to monitor the k8s node filesystem by using the metricbeat 7.2 deployment:

elastic/beats/blob/master/deploy/kubernetes/metricbeat-kubernetes.yaml

I have adapted a little bit the drop event to drop the container filesystem:

  processors:
  - drop_event.when.regexp:
      system.filesystem.mount_point: '^/(hostfs/)?(sys|cgroup|proc|dev|etc|host|lib)($|/)'
  - drop_event.when.regexp:
      system.filesystem.mount_point: '^/$'

And I have mount the node root path to the container into "/hostfs" (following https://www.elastic.co/guide/en/beats/metricbeat/current/running-on-docker.html#monitoring-host):

        - name: root
          mountPath: /hostfs
          readOnly: true 

I got the following result:

/hostfs	/dev/sda2	
/hostfs/boot	/dev/sda1
/hostfs/var/lib/kubelet/plugins/kubernetes.io/azure-disk/mounts/m3444057142	/dev/sdh	
/hostfs/var/lib/kubelet/plugins/kubernetes.io/azure-disk/mounts/m1430128950	/dev/sdg	
/hostfs/var/lib/kubelet/plugins/kubernetes.io/azure-disk/mounts/m3444057142	/dev/sdh	
/hostfs/var/lib/kubelet/plugins/kubernetes.io/azure-disk/mounts/m1524355945	/dev/sdi	
/hostfs/mnt/resource	/dev/sdb1

The issue is that the node filesystem has a mount on /var but it is not reported by metricbeat

A cat of mtab from the container (kubectl -n kube-system exec -it metricbeat-24qrc -- cat /etc/mtab) returns:

rootfs / rootfs rw 0 0
/dev/sdc1 / xfs rw,seclabel,relatime,attr2,inode64,noquota 0 0
/dev/sda2 /hostfs xfs ro,seclabel,relatime,attr2,inode64,noquota 0 0
/dev/sdc1 /hostfs/var xfs rw,seclabel,relatime,attr2,inode64,noquota 0 0
/dev/sdg /hostfs/var/lib/kubelet/plugins/kubernetes.io/azure-disk/mounts/m1430128950 ext4 rw,seclabel,relatime,data=ordered 0 0
/dev/sdh /hostfs/var/lib/kubelet/plugins/kubernetes.io/azure-disk/mounts/m3444057142 ext4 rw,seclabel,relatime,data=ordered 0 0
/dev/sdh /hostfs/var/lib/kubelet/pods/fafc3b8b-a74b-11e9-8de9-000d3a291e50/volumes/kubernetes.io~azure-disk/pvc-azertyu-ssss-ssss-azert-azerty ext4 rw,seclabel,relatime,data=ordered 0 0

var is well returned. What could possibly be wrong?

Thanks!

[Nicolas_Labrot]

I guess it is because of those statements: https://github.com/elastic/beats/blob/master/metricbeat/module/system/filesystem/helper.go#L93
"/" of the container and "/hostfs/var" of the host are on the same device.

Would it be possible to add a configuration to let the user choose if he wants to filter?