Filesystem metricset not reporting nfs and some host filesystems

Elastic Stack Beats
1

Continuing the discussion from Filesystem metricset not reporting nfs and other host filesystems:

we have the same problem on our environment (7.1 elk and stack).

Some info regarding this issue ?

2

The problem seems to be for specific version nfs4.

For the other nfs we added

filesystem.ignore_types: [autofs]

and we received the stats.

3

Hello Mario,
Would be possible to know:

  • the content of the file /etc/mtab (just the nfs mount is enough)?
  • the content of /etc/fstab (to know the mount parameters of your nfs4)?

It would help in order to reproduce the issue.

For your information, gosigar is used behind the scenes in Metricbeat, which is based on an open source project by Cloudfoundry.

4

For the specific entries

/etc/fstab
10.10.103.98:/penn_kahadb /var/activemq/data/kahadb nfs defaults 0 0
/etc/mtab
10.10.103.98:/penn_kahadb /var/activemq/data/kahadb nfs4 rw,relatime,vers=4.0,rsize=524288,wsize=524288,namlen=255,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,clie
ntaddr=10.10.103.55,local_lock=none,addr=10.10.103.98 0 0

5

Hello,
I've tried to reproduce setting up a dummy NFS share.
Tested on Metricbeat 7.2.1.

I've edited /etc/fstab and ran sudo mount -a:

192.168.1.206:/var/nfs/general /var/activemq/data/kahadb nfs4 defaults 0 0

It gets mounted as NFSv4.2:

192.168.1.206:/var/nfs/general /var/activemq/data/kahadb nfs4 rw,relatime,vers=4.2,rsize=131072,wsize=131072,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.1.205,local_lock=none,addr=192.168.1.206 0 0

The system module on Metricbeat is configured with:

- module: system
  period: 10s
  metricsets:
    - filesystem
    - fsstat
  filesystem.ignore_types: []

I get the following metrics:

{"@timestamp":"2019-07-31T23:56:07.082Z","@metadata":{"beat":"metricbeat","type":"_doc","version":"7.2.1"},"event":{"dataset":"system.filesystem","module":"system","duration":18487837},"metricset":{"name":"filesystem"},"service":{"type":"system"},"host":{"architecture":"x86_64","os":{"codename":"bionic","platform":"ubuntu","version":"18.04.2 LTS (Bionic Beaver)","family":"debian","name":"Ubuntu","kernel":"4.15.0-54-generic"},"id":"d545538ba5a549b284a3a9d9134bcea5","containerized":false,"hostname":"vm-VirtualBox","name":"vm-VirtualBox"},"agent":{"version":"7.2.1","type":"metricbeat","ephemeral_id":"bc0a6262-cb80-447e-8e51-b0151749ced8","hostname":"vm-VirtualBox","id":"e6b3f5d1-66e1-4d43-902d-23043b2b106e"},"ecs":{"version":"1.0.0"},"system":{"filesystem":{"used":{"pct":0.6336,"bytes":2489188352},"mount_point":"/var/activemq/data/kahadb","total":4160487424,"files":262144,"free_files":187464,"device_name":"192.168.1.206:/var/nfs/general","free":1671299072,"available":1439694848,"type":"nfs4"}}}

When forcing version NFSv4.0 using:

192.168.1.206:/var/nfs/general /var/activemq/data/kahadb nfs nfsvers=4.0 0 0

Associated /etc/mtab:

192.168.1.206:/var/nfs/general /var/activemq/data/kahadb nfs4 rw,relatime,vers=4.0,rsize=131072,wsize=131072,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.1.205,local_lock=none,addr=192.168.1.206 0 0

I still get the metrics:

{"@timestamp":"2019-08-01T00:02:19.743Z","@metadata":{"beat":"metricbeat","type":"_doc","version":"7.2.1"},"service":{"type":"system"},"system":{"filesystem":{"mount_point":"/var/activemq/data/kahadb","available":1439694848,"type":"nfs4","total":4160487424,"free":1671299072,"files":262144,"free_files":187465,"used":{"bytes":2489188352,"pct":0.6336},"device_name":"192.168.1.206:/var/nfs/general"}},"host":{"name":"vm-VirtualBox","hostname":"vm-VirtualBox","architecture":"x86_64","os":{"codename":"bionic","platform":"ubuntu","version":"18.04.2 LTS (Bionic Beaver)","family":"debian","name":"Ubuntu","kernel":"4.15.0-54-generic"},"id":"d545538ba5a549b284a3a9d9134bcea5","containerized":false},"agent":{"id":"e6b3f5d1-66e1-4d43-902d-23043b2b106e","version":"7.2.1","type":"metricbeat","ephemeral_id":"0be208ff-9ab5-4707-be38-e3403008e449","hostname":"vm-VirtualBox"},"ecs":{"version":"1.0.0"},"event":{"dataset":"system.filesystem","module":"system","duration":11436422},"metricset":{"name":"filesystem"}}

The filesystem.ignore_types is generated dynamically depending on the Operating System settings (code). In particular, it is reading procfs at /proc/filesystems, excluding the fs types nodev.

This is documented at https://www.elastic.co/guide/en/beats/metricbeat/6.7/metricbeat-metricset-system-filesystem.html#_configuration_5

In my case is:

nodev	sysfs
nodev	rootfs
nodev	ramfs
nodev	bdev
nodev	proc
nodev	cpuset
nodev	cgroup
nodev	cgroup2
nodev	tmpfs
nodev	devtmpfs
nodev	configfs
nodev	debugfs
nodev	tracefs
nodev	securityfs
nodev	sockfs
nodev	dax
nodev	bpf
nodev	pipefs
nodev	hugetlbfs
nodev	devpts
       	ext3
       	ext2
       	ext4
       	squashfs
       	vfat
nodev	ecryptfs
       	fuseblk
nodev	fuse
nodev	fusectl
nodev	pstore
nodev	efivarfs
nodev	mqueue
nodev	autofs
nodev	vboxsf
nodev	overlay
nodev	aufs
     	iso9660
     	xfs
     	jfs
     	msdos
     	ntfs
     	minix
     	hfs
     	hfsplus
     	qnx4
     	ufs
     	btrfs
nodev	rpc_pipefs
nodev	nfsd
nodev	nfs
nodev	nfs4
nodev	binfmt_misc
6

Hi Luca
can you add a second filesystem on same machine and verify again ?

7

Another NFS share? Exposed by the same host or a different one?

I've tested out:

{"@timestamp":"2019-08-01T17:36:07.972Z","@metadata":{"beat":"metricbeat","type":"_doc","version":"7.2.1"},"event":{"module":"system","duration":14673459,"dataset":"system.filesystem"},"metricset":{"name":"filesystem"},"service":{"type":"system"},"system":{"filesystem":{"available":1422786560,"used":{"bytes":2506227712,"pct":0.6379},"type":"nfs4","free":1654259712,"total":4160487424,"free_files":187456,"mount_point":"/var/activemq/data/kahadb","files":262144,"device_name":"192.168.1.206:/var/nfs/general"}},"host":{"name":"vm-VirtualBox","id":"d545538ba5a549b284a3a9d9134bcea5","containerized":false,"hostname":"vm-VirtualBox","architecture":"x86_64","os":{"family":"debian","name":"Ubuntu","kernel":"4.15.0-54-generic","codename":"bionic","platform":"ubuntu","version":"18.04.2 LTS (Bionic Beaver)"}},"agent":{"hostname":"vm-VirtualBox","id":"e6b3f5d1-66e1-4d43-902d-23043b2b106e","version":"7.2.1","type":"metricbeat","ephemeral_id":"7946d3bb-e3a6-45a6-af34-a8954c7b3395"},"ecs":{"version":"1.0.0"}}
{"@timestamp":"2019-08-01T17:36:07.972Z","@metadata":{"beat":"metricbeat","type":"_doc","version":"7.2.1"},"host":{"name":"vm-VirtualBox","os":{"family":"debian","name":"Ubuntu","kernel":"4.15.0-54-generic","codename":"bionic","platform":"ubuntu","version":"18.04.2 LTS (Bionic Beaver)"},"id":"d545538ba5a549b284a3a9d9134bcea5","containerized":false,"hostname":"vm-VirtualBox","architecture":"x86_64"},"system":{"filesystem":{"type":"nfs4","device_name":"192.168.1.206:/var/nfs/specific","total":4160487424,"free":1654259712,"files":262144,"free_files":187456,"used":{"bytes":2506227712,"pct":0.6379},"mount_point":"/var/activemq/data/kahadc","available":1422786560}},"event":{"module":"system","duration":14990521,"dataset":"system.filesystem"},"metricset":{"name":"filesystem"},"service":{"type":"system"},"agent":{"type":"metricbeat","ephemeral_id":"7946d3bb-e3a6-45a6-af34-a8954c7b3395","hostname":"vm-VirtualBox","id":"e6b3f5d1-66e1-4d43-902d-23043b2b106e","version":"7.2.1"},"ecs":{"version":"1.0.0"}}

Works with multiple NFS shares.

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.