I am using the latest Metricbeat from GitHub, cross compiled for use on aarch64. I have a device that was submitting default system statistics to Elastic just fine on Friday. Monday, it was not showing on the System Overview dashboard despite attempting to remake the dashboard with # ./metricbeat setup after clearing indices. Tuesday (this morning) I found that it was in fact submitting data up until a mysterious power cut, and hasn't been working at all since, despite the Metricbeat client seemingly showing success for every update.
Relevant areas of metricbeat.yml:
# =================================== Kibana ===================================
# Starting with Beats version 6.0.0, the dashboards are loaded via the Kibana API.
# This requires a Kibana endpoint configuration.
setup.kibana:
# Kibana Host
# Scheme and port can be left out and will be set to the default (http and 5601)
# In case you specify and additional path, the scheme is required: http://localhost:5601/path
# IPv6 addresses should always be defined as: https://[2001:db8::1]:5601
host: "172.24.0.201:5601"
# Kibana Space ID
# ID of the Kibana Space into which the dashboards should be loaded. By default,
# the Default Space will be used.
#space.id:
# ================================== Outputs ===================================
# Configure what output to use when sending the data collected by the beat.
# ---------------------------- Elasticsearch Output ----------------------------
output.elasticsearch:
# Array of hosts to connect to.
hosts: ["172.24.0.201:9200"]
# Protocol - either `http` (default) or `https`.
#protocol: "https"
# Authentication credentials - either API key or username/password.
#api_key: "id:api_key"
#username: "elastic"
#password: "changeme"
# ------------------------------ Logstash Output -------------------------------
#output.logstash:
# The Logstash hosts
#hosts: ["192.168.201.10:5044"]
# Optional SSL. By default is off.
# List of root certificates for HTTPS server verifications
#ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]
# Certificate for SSL client authentication
#ssl.certificate: "/etc/pki/client/cert.pem"
# Client Certificate Key
#ssl.key: "/etc/pki/client/cert.key"
# ================================= Processors =================================
# Configure processors to enhance or manipulate events generated by the beat.
processors:
- add_host_metadata: ~
- add_cloud_metadata: ~
- add_docker_metadata: ~
- add_kubernetes_metadata: ~
# ./metricbeat run -e:
2020-06-23T10:23:52.153Z INFO instance/beat.go:628 Home path: [/root/metricbeat] Config path: [/root/metricbeat] Data path: [/root/metricbeat/data] Logs path: [/root/metricbeat/logs]
2020-06-23T10:23:52.153Z INFO instance/beat.go:636 Beat ID: e0b4bd2a-551c-4b20-b1ac-0318b7c32023
2020-06-23T10:23:52.163Z INFO [seccomp] seccomp/seccomp.go:124 Syscall filter successfully installed
2020-06-23T10:23:52.163Z INFO [beat] instance/beat.go:964 Beat info {"system_info": {"beat": {"path": {"config": "/root/metricbeat", "data": "/root/metricbeat/data", "home": "/root/metricbeat", "logs": "/root/metricbeat/logs"}, "type": "metricbeat", "uuid": "e0b4bd2a-551c-4b20-b1ac-0318b7c32023"}}}
2020-06-23T10:23:52.163Z INFO [beat] instance/beat.go:973 Build info {"system_info": {"build": {"commit": "unknown", "libbeat": "8.0.0", "time": "1754-08-30T22:43:41.128Z", "version": "8.0.0"}}}
2020-06-23T10:23:52.164Z INFO [beat] instance/beat.go:976 Go runtime info {"system_info": {"go": {"os":"linux","arch":"arm","max_procs":4,"version":"go1.14.4"}}}
2020-06-23T10:23:52.167Z INFO [beat] instance/beat.go:980 Host info {"system_info": {"host": {"architecture":"aarch64","boot_time":"2020-06-22T16:36:36Z","containerized":false,"name":"alpine-pi","ip":["127.0.0.1/8","::1/128","172.24.0.210/24","10.0.0.74/24","fe80::c000:566e:ec74:e3c0/64"],"kernel_version":"4.14.0","mac":["02:01:fb:98:b6:29"],"os":{"family":"","platform":"alpine","name":"Alpine Linux","version":"","major":0,"minor":0,"patch":0},"timezone":"UTC","timezone_offset_sec":0}}}
2020-06-23T10:23:52.170Z INFO [beat] instance/beat.go:1009 Process info {"system_info": {"process": {"capabilities": {"inheritable":null,"permitted":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read"],"effective":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read"],"bounding":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read"],"ambient":null}, "cwd": "/root/metricbeat", "exe": "/root/metricbeat/metricbeat", "name": "metricbeat", "pid": 6650, "ppid": 6138, "seccomp": {"mode":"filter","no_new_privs":true}, "start_time": "2020-06-23T10:23:51.060Z"}}}
2020-06-23T10:23:52.171Z INFO instance/beat.go:298 Setup Beat: metricbeat; Version: 8.0.0
2020-06-23T10:23:52.171Z INFO [index-management] idxmgmt/std.go:184 Set output.elasticsearch.index to 'metricbeat-8.0.0' as ILM is enabled.
2020-06-23T10:23:52.172Z INFO eslegclient/connection.go:99 elasticsearch url: http://172.24.0.201:9200
2020-06-23T10:23:52.173Z INFO [publisher] pipeline/module.go:113 Beat name: alpine-pi
2020-06-23T10:23:52.303Z ERROR [registry.lightmodules] mb/lightmodules.go:147 Failed to list light metricsets for module cloudfoundry: getting metricsets for module 'cloudfoundry': loading metric sets for light module 'cloudfoundry': loading light metricset 'counter': loading metricset manifest from '/root/metricbeat/module/cloudfoundry/counter/manifest.yml': stat /root/metricbeat/module/cloudfoundry/counter/manifest.yml: no such file or directory
2020-06-23T10:23:52.326Z INFO [monitoring] log/log.go:118 Starting metrics logging every 30s
2020-06-23T10:23:52.327Z INFO instance/beat.go:449 metricbeat start running.
2020-06-23T10:23:52.332Z INFO filesystem/filesystem.go:57 Ignoring filesystem types: sysfs, rootfs, ramfs, bdev, proc, cpuset, cgroup, cgroup2, tmpfs, devtmpfs, binfmt_misc, configfs, debugfs,
securityfs, sockfs, dax, pipefs, hugetlbfs, rpc_pipefs, devpts, nfs, nfs4, nfsd, autofs, mqueue
2020-06-23T10:23:52.333Z INFO [system.fsstat] fsstat/fsstat.go:56 Ignoring filesystem types: %ssysfs, rootfs, ramfs, bdev, proc, cpuset, cgroup, cgroup2, tmpfs, devtmpfs, binfmt_misc, configfs, debugfs, securityfs, sockfs, dax, pipefs, hugetlbfs, rpc_pipefs, devpts, nfs, nfs4, nfsd, autofs, mqueue
2020-06-23T10:23:52.334Z INFO cfgfile/reload.go:164 Config reloader started
2020-06-23T10:23:52.341Z INFO filesystem/filesystem.go:57 Ignoring filesystem types: sysfs, rootfs, ramfs, bdev, proc, cpuset, cgroup, cgroup2, tmpfs, devtmpfs, binfmt_misc, configfs, debugfs,
securityfs, sockfs, dax, pipefs, hugetlbfs, rpc_pipefs, devpts, nfs, nfs4, nfsd, autofs, mqueue
2020-06-23T10:23:52.342Z INFO [system.fsstat] fsstat/fsstat.go:56 Ignoring filesystem types: %ssysfs, rootfs, ramfs, bdev, proc, cpuset, cgroup, cgroup2, tmpfs, devtmpfs, binfmt_misc, configfs, debugfs, securityfs, sockfs, dax, pipefs, hugetlbfs, rpc_pipefs, devpts, nfs, nfs4, nfsd, autofs, mqueue
2020-06-23T10:23:52.349Z INFO cfgfile/reload.go:224 Loading of config files completed.
2020-06-23T10:23:55.159Z INFO [add_cloud_metadata] add_cloud_metadata/add_cloud_metadata.go:89 add_cloud_metadata: hosting provider type not detected.
2020-06-23T10:23:56.163Z INFO [publisher_pipeline_output] pipeline/output.go:143 Connecting to backoff(elasticsearch(http://172.24.0.201:9200))
2020-06-23T10:23:56.165Z INFO [publisher] pipeline/retry.go:219 retryer: send unwait signal to consumer
2020-06-23T10:23:56.167Z INFO [publisher] pipeline/retry.go:223 done
2020-06-23T10:23:56.174Z INFO [esclientleg] eslegclient/connection.go:314 Attempting to connect to Elasticsearch version 7.8.0
2020-06-23T10:23:56.177Z INFO [esclientleg] eslegclient/connection.go:314 Attempting to connect to Elasticsearch version 7.8.0
2020-06-23T10:23:56.223Z INFO [index-management] idxmgmt/std.go:261 Auto ILM enable success.
2020-06-23T10:23:56.225Z INFO [index-management.ilm] ilm/std.go:139 do not generate ilm policy: exists=true, overwrite=false
2020-06-23T10:23:56.225Z INFO [index-management] idxmgmt/std.go:274 ILM policy successfully loaded.
2020-06-23T10:23:56.225Z INFO [index-management] idxmgmt/std.go:407 Set setup.template.name to '{metricbeat-8.0.0 {now/d}-000001}' as ILM is enabled.
2020-06-23T10:23:56.225Z INFO [index-management] idxmgmt/std.go:412 Set setup.template.pattern to 'metricbeat-8.0.0-*' as ILM is enabled.
2020-06-23T10:23:56.226Z INFO [index-management] idxmgmt/std.go:446 Set settings.index.lifecycle.rollover_alias in template to {metricbeat-8.0.0 {now/d}-000001} as ILM is enabled.
2020-06-23T10:23:56.226Z INFO [index-management] idxmgmt/std.go:450 Set settings.index.lifecycle.name in template to {metricbeat {"policy":{"phases":{"hot":{"actions":{"rollover":{"max_age":"30d","max_size":"50gb"}}}}}}} as ILM is enabled.
2020-06-23T10:23:56.229Z INFO [template_loader] template/load.go:101 Template metricbeat-8.0.0 already exists and will not be overwritten.
2020-06-23T10:23:56.229Z INFO [index-management] idxmgmt/std.go:298 Loaded index template.
2020-06-23T10:23:56.232Z INFO [index-management] idxmgmt/std.go:309 Write alias successfully generated.
2020-06-23T10:23:56.235Z INFO [publisher_pipeline_output] pipeline/output.go:151 Connection to backoff(elasticsearch(http://172.24.0.201:9200)) established
# ./metricbeat test config, # ./metricbeat test modules, and # ./metricbeat test output all show OK. Any ideas?
(CPU usage for the misbehaving system. All other statistics cut off at the same time, and all gauges at the top are 0)
(where
alpine-bin is the same executable on a separate system, Server-One and laptop are rpm installs, and alpine-pi is the hostname of the misbehaving system)