KVM module, dial: unknown network

[Takk]

Hello everyone!

It seems that I have a problem while using KVM module. I cannot collect any metric from KVM.

Here the /var/log/metricbeat/metricbeat log:

2018-07-29T11:35:23.412+0200 INFO elasticsearch/client.go:145 Elasticsearch url: http://localhost:9200
2018-07-29T11:35:23.413+0200 INFO pipeline/module.go:81 Beat name: alex-VirtualBox
2018-07-29T11:35:23.414+0200 INFO [monitoring] log/log.go:97 Starting metrics logging every 30s
2018-07-29T11:35:23.415+0200 INFO elasticsearch/client.go:145 Elasticsearch url: http://localhost:9200
2018-07-29T11:35:23.420+0200 INFO elasticsearch/client.go:690 Connected to Elasticsearch version 6.3.0
2018-07-29T11:35:23.420+0200 INFO kibana/client.go:90 Kibana url: http://localhost:5601
2018-07-29T11:35:38.338+0200 INFO instance/beat.go:607 Kibana dashboards successfully loaded.
2018-07-29T11:35:38.338+0200 INFO instance/beat.go:315 metricbeat start running.
2018-07-29T11:35:38.339+0200 WARN [cfgwarn] dommemstat/dommemstat.go:48 EXPERIMENTAL: The kvm dommemstat metricset is experimental.
2018-07-29T11:35:38.340+0200 INFO filesystem/filesystem.go:41 Ignoring filesystem types: sysfs, rootfs, ramfs, bdev, proc, cpuset, cgroup, cgroup2, tmpfs, devtmpfs, configfs, debugfs, tracefs, securityfs, sockfs, dax, bpf, pipefs, hugetlbfs, devpts, ecryptfs, fuse, fusectl, pstore, mqueue, autofs, vboxsf
2018-07-29T11:35:38.340+0200 INFO fsstat/fsstat.go:42 Ignoring filesystem types: sysfs, rootfs, ramfs, bdev, proc, cpuset, cgroup, cgroup2, tmpfs, devtmpfs, configfs, debugfs, tracefs, securityfs, sockfs, dax, bpf, pipefs, hugetlbfs, devpts, ecryptfs, fuse, fusectl, pstore, mqueue, autofs, vboxsf
2018-07-29T11:35:38.340+0200 INFO cfgfile/reload.go:122 Config reloader started
2018-07-29T11:35:38.341+0200 WARN [cfgwarn] dommemstat/dommemstat.go:48 EXPERIMENTAL: The kvm dommemstat metricset is experimental.
2018-07-29T11:35:38.341+0200 INFO filesystem/filesystem.go:41 Ignoring filesystem types: sysfs, rootfs, ramfs, bdev, proc, cpuset, cgroup, cgroup2, tmpfs, devtmpfs, configfs, debugfs, tracefs, securityfs, sockfs, dax, bpf, pipefs, hugetlbfs, devpts, ecryptfs, fuse, fusectl, pstore, mqueue, autofs, vboxsf
2018-07-29T11:35:38.342+0200 INFO fsstat/fsstat.go:42 Ignoring filesystem types: sysfs, rootfs, ramfs, bdev, proc, cpuset, cgroup, cgroup2, tmpfs, devtmpfs, configfs, debugfs, tracefs, securityfs, sockfs, dax, bpf, pipefs, hugetlbfs, devpts, ecryptfs, fuse, fusectl, pstore, mqueue, autofs, vboxsf
2018-07-29T11:35:38.342+0200 INFO cfgfile/reload.go:253 Starting 4 runners ...
2018-07-29T11:35:38.342+0200 INFO cfgfile/reload.go:214 Loading of config files completed.
2018-07-29T11:35:38.342+0200 ERROR runtime/panic.go:63 recovered from panic while fetching 'kvm/dommemstat' for host 'localhost'. Recovering, but please report this. {"panic": "runtime error: invalid memory address or nil pointer dereference", "stack": "github.com/elastic/beats/libbeat/logp.Recover\n\t/go/src/github.com/elastic/beats/libbeat/logp/global.go:88\nruntime.call32\n\t/usr/local/go/src/runtime/asm_amd64.s:509\nruntime.gopanic\n\t/usr/local/go/src/runtime/panic.go:491\nruntime.panicmem\n\t/usr/local/go/src/runtime/panic.go:63\nruntime.sigpanic\n\t/usr/local/go/src/runtime/signal_unix.go:367\ngithub.com/elastic/beats/metricbeat/module/kvm/dommemstat.(*MetricSet).Fetch\n\t/go/src/github.com/elastic/beats/metricbeat/module/kvm/dommemstat/dommemstat.go:89\ngithub.com/elastic/beats/metricbeat/mb/module.(*metricSetWrapper).fetch\n\t/go/src/github.com/elastic/beats/metricbeat/mb/module/wrapper.go:213\ngithub.com/elastic/beats/metricbeat/mb/module.(*metricSetWrapper).startPeriodicFetching\n\t/go/src/github.com/elastic/beats/metricbeat/mb/module/wrapper.go:184\ngithub.com/elastic/beats/metricbeat/mb/module.(*metricSetWrapper).run\n\t/go/src/github.com/elastic/beats/metricbeat/mb/module/wrapper.go:171\ngithub.com/elastic/beats/metricbeat/mb/module.(*Wrapper).Start.func1\n\t/go/src/github.com/elastic/beats/metricbeat/mb/module/wrapper.go:112"}

And on Kibana, I get this document, with the error message "dial: unknown network":
{ "_index": "metricbeat-6.3.2-2018.07.29", "_type": "doc", "_id": "a5Jq5WQBagAcT8VpKu8h", "_version": 1, "_score": null, "_source": { "@timestamp": "2018-07-29T09:41:36.406Z", "error": { "message": "dial: unknown network " }, "metricset": { "rtt": 6, "name": "dommemstat", "module": "kvm", "host": "localhost" }, "beat": { "name": "testsrv", "hostname": "testsrv", "version": "6.3.2" }, "host": { "name": "testsrv" } }, "fields": { "@timestamp": [ "2018-07-29T09:41:36.406Z" ] }, "sort": [ 1532857296406 ] }

Did someone have the same problem?
Thanks,

Alex

[adrisr (Adrian Serrano)]

Hi,

Two issues here, one is the panic, for which I already submitted a fix.

And the real issue is that it cannot connect to libvirtd.

Is your libvirtd running?

I think that the default configuration for the kvm module may not be correct. Unfortunately, I don't have a kvm/libvirt setup so I can't test it.

Try editting the /etc/metricbeat/modules.d/kvm.conf file and changing the line:

hosts: ["localhost"]

to

hosts: ["unix:///var/run/libvirt/libvirt-sock"]

or

hosts: ["tcp://localhost:16509"]

[Takk]

Hi,

Awesome! Both of the solutions works perfectly.
Thanks a lot!

For those who struggle with TCP, do not forget to enable the listening option on libvirtd deamon by adding the following line in the file /etc/default/libvirtd:

libvirtd_opts="-l"