Continuing the discussion from Filesystem metricset not reporting nfs and other host filesystems:
we have the same problem on our environment (7.1 elk and stack).
Some info regarding this issue ?
The problem seems to be for specific version nfs4.
For the other nfs we added
filesystem.ignore_types: [autofs]
and we received the stats.
Hello Mario,
Would be possible to know:
- the content of the file
/etc/mtab(just thenfsmount is enough)? - the content of
/etc/fstab(to know the mount parameters of yournfs4)?
It would help in order to reproduce the issue.
For your information, gosigar is used behind the scenes in Metricbeat, which is based on an open source project by Cloudfoundry.
For the specific entries
/etc/fstab
10.10.103.98:/penn_kahadb /var/activemq/data/kahadb nfs defaults 0 0
/etc/mtab
10.10.103.98:/penn_kahadb /var/activemq/data/kahadb nfs4 rw,relatime,vers=4.0,rsize=524288,wsize=524288,namlen=255,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,clie
ntaddr=10.10.103.55,local_lock=none,addr=10.10.103.98 0 0
Hello,
I've tried to reproduce setting up a dummy NFS share.
Tested on Metricbeat 7.2.1.
I've edited /etc/fstab and ran sudo mount -a:
192.168.1.206:/var/nfs/general /var/activemq/data/kahadb nfs4 defaults 0 0
It gets mounted as NFSv4.2:
192.168.1.206:/var/nfs/general /var/activemq/data/kahadb nfs4 rw,relatime,vers=4.2,rsize=131072,wsize=131072,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.1.205,local_lock=none,addr=192.168.1.206 0 0
The system module on Metricbeat is configured with:
- module: system
period: 10s
metricsets:
- filesystem
- fsstat
filesystem.ignore_types: []
I get the following metrics:
{"@timestamp":"2019-07-31T23:56:07.082Z","@metadata":{"beat":"metricbeat","type":"_doc","version":"7.2.1"},"event":{"dataset":"system.filesystem","module":"system","duration":18487837},"metricset":{"name":"filesystem"},"service":{"type":"system"},"host":{"architecture":"x86_64","os":{"codename":"bionic","platform":"ubuntu","version":"18.04.2 LTS (Bionic Beaver)","family":"debian","name":"Ubuntu","kernel":"4.15.0-54-generic"},"id":"d545538ba5a549b284a3a9d9134bcea5","containerized":false,"hostname":"vm-VirtualBox","name":"vm-VirtualBox"},"agent":{"version":"7.2.1","type":"metricbeat","ephemeral_id":"bc0a6262-cb80-447e-8e51-b0151749ced8","hostname":"vm-VirtualBox","id":"e6b3f5d1-66e1-4d43-902d-23043b2b106e"},"ecs":{"version":"1.0.0"},"system":{"filesystem":{"used":{"pct":0.6336,"bytes":2489188352},"mount_point":"/var/activemq/data/kahadb","total":4160487424,"files":262144,"free_files":187464,"device_name":"192.168.1.206:/var/nfs/general","free":1671299072,"available":1439694848,"type":"nfs4"}}}
When forcing version NFSv4.0 using:
192.168.1.206:/var/nfs/general /var/activemq/data/kahadb nfs nfsvers=4.0 0 0
Associated /etc/mtab:
192.168.1.206:/var/nfs/general /var/activemq/data/kahadb nfs4 rw,relatime,vers=4.0,rsize=131072,wsize=131072,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.1.205,local_lock=none,addr=192.168.1.206 0 0
I still get the metrics:
{"@timestamp":"2019-08-01T00:02:19.743Z","@metadata":{"beat":"metricbeat","type":"_doc","version":"7.2.1"},"service":{"type":"system"},"system":{"filesystem":{"mount_point":"/var/activemq/data/kahadb","available":1439694848,"type":"nfs4","total":4160487424,"free":1671299072,"files":262144,"free_files":187465,"used":{"bytes":2489188352,"pct":0.6336},"device_name":"192.168.1.206:/var/nfs/general"}},"host":{"name":"vm-VirtualBox","hostname":"vm-VirtualBox","architecture":"x86_64","os":{"codename":"bionic","platform":"ubuntu","version":"18.04.2 LTS (Bionic Beaver)","family":"debian","name":"Ubuntu","kernel":"4.15.0-54-generic"},"id":"d545538ba5a549b284a3a9d9134bcea5","containerized":false},"agent":{"id":"e6b3f5d1-66e1-4d43-902d-23043b2b106e","version":"7.2.1","type":"metricbeat","ephemeral_id":"0be208ff-9ab5-4707-be38-e3403008e449","hostname":"vm-VirtualBox"},"ecs":{"version":"1.0.0"},"event":{"dataset":"system.filesystem","module":"system","duration":11436422},"metricset":{"name":"filesystem"}}
The filesystem.ignore_types is generated dynamically depending on the Operating System settings (code). In particular, it is reading procfs at /proc/filesystems, excluding the fs types nodev.
This is documented at https://www.elastic.co/guide/en/beats/metricbeat/6.7/metricbeat-metricset-system-filesystem.html#_configuration_5
In my case is:
nodev sysfs
nodev rootfs
nodev ramfs
nodev bdev
nodev proc
nodev cpuset
nodev cgroup
nodev cgroup2
nodev tmpfs
nodev devtmpfs
nodev configfs
nodev debugfs
nodev tracefs
nodev securityfs
nodev sockfs
nodev dax
nodev bpf
nodev pipefs
nodev hugetlbfs
nodev devpts
ext3
ext2
ext4
squashfs
vfat
nodev ecryptfs
fuseblk
nodev fuse
nodev fusectl
nodev pstore
nodev efivarfs
nodev mqueue
nodev autofs
nodev vboxsf
nodev overlay
nodev aufs
iso9660
xfs
jfs
msdos
ntfs
minix
hfs
hfsplus
qnx4
ufs
btrfs
nodev rpc_pipefs
nodev nfsd
nodev nfs
nodev nfs4
nodev binfmt_misc
Hi Luca
can you add a second filesystem on same machine and verify again ?
Another NFS share? Exposed by the same host or a different one?
I've tested out:
{"@timestamp":"2019-08-01T17:36:07.972Z","@metadata":{"beat":"metricbeat","type":"_doc","version":"7.2.1"},"event":{"module":"system","duration":14673459,"dataset":"system.filesystem"},"metricset":{"name":"filesystem"},"service":{"type":"system"},"system":{"filesystem":{"available":1422786560,"used":{"bytes":2506227712,"pct":0.6379},"type":"nfs4","free":1654259712,"total":4160487424,"free_files":187456,"mount_point":"/var/activemq/data/kahadb","files":262144,"device_name":"192.168.1.206:/var/nfs/general"}},"host":{"name":"vm-VirtualBox","id":"d545538ba5a549b284a3a9d9134bcea5","containerized":false,"hostname":"vm-VirtualBox","architecture":"x86_64","os":{"family":"debian","name":"Ubuntu","kernel":"4.15.0-54-generic","codename":"bionic","platform":"ubuntu","version":"18.04.2 LTS (Bionic Beaver)"}},"agent":{"hostname":"vm-VirtualBox","id":"e6b3f5d1-66e1-4d43-902d-23043b2b106e","version":"7.2.1","type":"metricbeat","ephemeral_id":"7946d3bb-e3a6-45a6-af34-a8954c7b3395"},"ecs":{"version":"1.0.0"}}
{"@timestamp":"2019-08-01T17:36:07.972Z","@metadata":{"beat":"metricbeat","type":"_doc","version":"7.2.1"},"host":{"name":"vm-VirtualBox","os":{"family":"debian","name":"Ubuntu","kernel":"4.15.0-54-generic","codename":"bionic","platform":"ubuntu","version":"18.04.2 LTS (Bionic Beaver)"},"id":"d545538ba5a549b284a3a9d9134bcea5","containerized":false,"hostname":"vm-VirtualBox","architecture":"x86_64"},"system":{"filesystem":{"type":"nfs4","device_name":"192.168.1.206:/var/nfs/specific","total":4160487424,"free":1654259712,"files":262144,"free_files":187456,"used":{"bytes":2506227712,"pct":0.6379},"mount_point":"/var/activemq/data/kahadc","available":1422786560}},"event":{"module":"system","duration":14990521,"dataset":"system.filesystem"},"metricset":{"name":"filesystem"},"service":{"type":"system"},"agent":{"type":"metricbeat","ephemeral_id":"7946d3bb-e3a6-45a6-af34-a8954c7b3395","hostname":"vm-VirtualBox","id":"e6b3f5d1-66e1-4d43-902d-23043b2b106e","version":"7.2.1"},"ecs":{"version":"1.0.0"}}
Works with multiple NFS shares.
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.