I installed the Metricbeat StatsD module a few days ago and it's reporting data on the machine it's installed on, but not reporting any custom metrics I send to it.
For example:
echo "test.metric:1|c" | nc -u -w0 <IP ADDRESS> 8125
That should log test.metric to the statsd server. However, when I go to Kibana cloud, it's not there.
The statsd module also doesn't have any documentation, so I'm not sure how to troubleshoot it.
Running the following command returns successful:
nc -v -u -z -w 3 <IP ADDRESS> 8125
Here are some of the metricbeat logs:
Dec 01 21:08:51 systemd[1]: Started Metricbeat is a lightweight shipper for metrics..
Dec 01 21:08:51 INFO instance/beat.go:665 Home path: [/usr/share/metricbeat] Config path: [/etc/metricbeat] Data path: [/var/lib/metricbeat] Logs path: [/var/log/metricbeat]
Dec 01 21:08:51 INFO instance/beat.go:673 Beat ID: 6a3c328a-2618-4f84-a16d-xxx
Dec 01 21:08:51 INFO [seccomp] seccomp/seccomp.go:124 Syscall filter successfully installed
Dec 01 21:08:51 INFO [beat] instance/beat.go:1014 Beat info {"system_info": {"beat": {"path": {"config": "/etc/metricbeat", "data": "/var/lib/metricbeat", "home": "/usr/share/metricbeat", "logs": "/var/log/metricbeat"}, "type": "metricbeat", "uuid": "6a3c328a-2618-4f84-a16d-263dab80cc9b"}}}
Dec 01 21:08:51 INFO [beat] instance/beat.go:1023 Build info {"system_info": {"build": {"commit": "fd322dad6ceafec40c84df4d2a0694ea357d16cc", "libbeat": "7.15.2", "time": "2021-11-04T14:35:34.000Z", "version": "7.15.2"}}}
Dec 01 21:08:51 INFO [beat] instance/beat.go:1026 Go runtime info {"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":1,"version":"go1.16.6"}}}
Dec 01 21:08:51 INFO [add_cloud_metadata] add_cloud_metadata/add_cloud_metadata.go:105 add_cloud_metadata: hosting provider type detected as aws, metadata={"cloud":{"account":{"id":"xxx"},"availability_zone":"us-west-1a","image":{"id":"ami-xxx"},"instance":{"id":"i-xxx"},"machine":{"type":"t2.micro"},"provider":"aws","region":"us-west-1","service":{"name":"EC2"}}}
Dec 01 21:08:51 INFO [beat] instance/beat.go:1030 Host info {"system_info": {"host": {"architecture":"x86_64","boot_time":"2021-12-01T21:08:20Z","containerized":false,"name":"ip-10-0-0-85.us-west-1.compute.internal","ip":["127.0.0.1/8","::1/128","10.0.0.85/24","fe80::4cd:25ff:fe5e:ebcf/64"],"kernel_version":"5.10.75-79.358.amzn2.x86_64","mac":["06:cd:25:5e:eb:cf"],"os":{"type":"linux","family":"redhat","platform":"amzn","name":"Amazon Linux","version":"2","major":2,"minor":0,"patch":0,"codename":"Karoo"},"timezone":"UTC","timezone_offset_sec":0,"id":"ec241e799eaf2c0ee222f87da91ab53c"}}}
Dec 01 21:08:51 INFO [beat] instance/beat.go:1059 Process info {"system_info": {"process": {"capabilities": {"inheritable":null,"permitted":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read","38","39","40"],"effective":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read","38","39","40"],"bounding":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read","38","39","40"],"ambient":null}, "cwd": "/", "exe": "/usr/share/metricbeat/bin/metricbeat", "name": "metricbeat", "pid": 3406, "ppid": 1, "seccomp": {"mode":"filter","no_new_privs":true}, "start_time": "2021-12-01T21:08:51.070Z"}}}
Dec 01 21:08:51 INFO instance/beat.go:309 Setup Beat: metricbeat; Version: 7.15.2
Dec 01 21:08:51 INFO [esclientleg] eslegclient/connection.go:100 elasticsearch url: xxx
Dec 01 21:08:51 INFO [publisher] pipeline/module.go:113 Beat name: ip-10-0-0-xx.us-west-1.compute.internal
Dec 01 21:08:51 INFO [monitoring] log/log.go:142 Starting metrics logging every 30s
Dec 01 21:08:51 INFO instance/beat.go:473 metricbeat start running.
Dec 01 21:08:51 INFO filesystem/filesystem.go:57 Ignoring filesystem types: sysfs, tmpfs, bdev, proc, cgroup, cgroup2, cpuset, devtmpfs, debugfs, tracefs, securityfs, sockfs, bpf, pipefs, ramfs, hugetlbfs, devpts, autofs, mqueue, pstore, rpc_pipefs
Dec 01 21:08:51 INFO [system.fsstat] fsstat/fsstat.go:57 Ignoring filesystem types: %ssysfs, tmpfs, bdev, proc, cgroup, cgroup2, cpuset, devtmpfs, debugfs, tracefs, securityfs, sockfs, bpf, pipefs, ramfs, hugetlbfs, devpts, autofs, mqueue, pstore, rpc_pipefs
Dec 01 21:08:51 INFO cfgfile/reload.go:164 Config reloader started
Dec 01 21:08:51 INFO filesystem/filesystem.go:57 Ignoring filesystem types: sysfs, tmpfs, bdev, proc, cgroup, cgroup2, cpuset, devtmpfs, debugfs, tracefs, securityfs, sockfs, bpf, pipefs, ramfs, hugetlbfs, devpts, autofs, mqueue, pstore, rpc_pipefs
Dec 01 21:08:51 INFO [system.fsstat] fsstat/fsstat.go:57 Ignoring filesystem types: %ssysfs, tmpfs, bdev, proc, cgroup, cgroup2, cpuset, devtmpfs, debugfs, tracefs, securityfs, sockfs, bpf, pipefs, ramfs, hugetlbfs, devpts, autofs, mqueue, pstore, rpc_pipefs
Dec 01 21:08:51 INFO cfgfile/reload.go:224 Loading of config files completed.
Dec 01 21:08:52 INFO [publisher_pipeline_output] pipeline/output.go:143 Connecting to backoff(elasticsearch(xxx))
Dec 01 21:08:52 INFO [publisher] pipeline/retry.go:219 retryer: send unwait signal to consumer
Dec 01 21:08:52 INFO [publisher] pipeline/retry.go:223 done
Dec 01 21:08:52 INFO [esclientleg] eslegclient/connection.go:273 Attempting to connect to Elasticsearch version 7.15.2
Dec 01 21:08:52 INFO [esclientleg] eslegclient/connection.go:273 Attempting to connect to Elasticsearch version 7.15.2
Dec 01 21:08:52 INFO template/load.go:111 Template "metricbeat-7.15.2" already exists and will not be overwritten.
Dec 01 21:08:52 INFO [index-management] idxmgmt/std.go:297 Loaded index template.
Dec 01 21:08:52 INFO [publisher_pipeline_output] pipeline/output.go:151 Connection to backoff(elasticsearch(xxx)) established
Dec 01 21:09:21 INFO [monitoring] log/log.go:184 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":80,"time":{"ms":88}},"total":{"ticks":410,"time":{"ms":423},"value":410},"user":{"ticks":330,"time":{"ms":335}}},"handles":{"limit":{"hard":65535,"soft":65535},"open":9},"info":{"ephemeral_id":"99378f36-751c-43b7-acf6-3ab5dfc80c04","uptime":{"ms":30294},"version":"7.15.2"},"memstats":{"gc_next":21175168,"memory_alloc":10962984,"memory_sys":75318280,"memory_total":69115864,"rss":151900160},"runtime":{"goroutines":61}},"libbeat":{"config":{"module":{"running":3,"starts":3},"reloads":1,"scans":1},"output":{"events":{"acked":53,"active":0,"batches":3,"total":53},"read":{"bytes":5728},"type":"elasticsearch","write":{"bytes":90504}},"pipeline":{"clients":10,"events":{"active":0,"published":53,"retry":20,"total":53},"queue":{"acked":53,"max_events":4096}}},"metricbeat":{"system":{"cpu":{"events":3,"success":3},"filesystem":{"events":1,"success":1},"fsstat":{"events":1,"success":1},"load":{"events":3,"success":3},"memory":{"events":3,"success":3},"network":{"events":8,"success":8},"process":{"events":27,"success":27},"process_summary":{"events":3,"success":3},"socket_summary":{"events":3,"success":3},"uptime":{"events":1,"success":1}}},"system":{"cpu":{"cores":1},"load":{"1":0.39,"15":0.05,"5":0.13,"norm":{"1":0.39,"15":0.05,"5":0.13}}}}}}
...
Dec 04 01:18:21 INFO [monitoring] log/log.go:184 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":331180,"time":{"ms":53}},"total":{"ticks":1011730,"time":{"ms":163},"value":1011730},"user":{"ticks":680550,"time":{"ms":110}}},"handles":{"limit":{"hard":65535,"soft":65535},"open":9},"info":{"ephemeral_id":"99378f36-751c-43b7-acf6-3ab5dfc80c04","uptime":{"ms":187770294},"version":"7.15.2"},"memstats":{"gc_next":21149056,"memory_alloc":10973888,"memory_total":175114131264,"rss":154943488},"runtime":{"goroutines":61}},"libbeat":{"config":{"module":{"running":3}},"output":{"events":{"acked":52,"active":0,"batches":3,"total":52},"read":{"bytes":2038},"write":{"bytes":87491}},"pipeline":{"clients":10,"events":{"active":0,"published":52,"total":52},"queue":{"acked":52}}},"metricbeat":{"system":{"cpu":{"events":3,"success":3},"filesystem":{"events":1,"success":1},"fsstat":{"events":1,"success":1},"load":{"events":3,"success":3},"memory":{"events":3,"success":3},"network":{"events":9,"success":9},"process":{"events":26,"success":26},"process_summary":{"events":3,"success":3},"socket_summary":{"events":3,"success":3}}},"system":{"load":{"1":0,"15":0,"5":0,"norm":{"1":0,"15":0,"5":0}}}}}}
Dec 04 01:18:51 INFO [monitoring] log/log.go:184 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":331220,"time":{"ms":44}},"total":{"ticks":1011890,"time":{"ms":167},"value":1011890},"user":{"ticks":680670,"time":{"ms":123}}},"handles":{"limit":{"hard":65535,"soft":65535},"open":9},"info":{"ephemeral_id":"99378f36-751c-43b7-acf6-3ab5dfc80c04","uptime":{"ms":187800296},"version":"7.15.2"},"memstats":{"gc_next":21241744,"memory_alloc":11189224,"memory_total":175142818352,"rss":154517504},"runtime":{"goroutines":61}},"libbeat":{"config":{"module":{"running":3}},"output":{"events":{"acked":49,"active":0,"batches":3,"total":49},"read":{"bytes":2009},"write":{"bytes":83066}},"pipeline":{"clients":10,"events":{"active":0,"published":49,"total":49},"queue":{"acked":49}}},"metricbeat":{"system":{"cpu":{"events":3,"success":3},"load":{"events":3,"success":3},"memory":{"events":3,"success":3},"network":{"events":9,"success":9} ,"process":{"events":25,"success":25},"process_summary":{"events":3,"success":3},"socket_summary":{"events":3,"success":3}}},"system":{"load":{"1":0,"15":0,"5":0,"norm":{"1":0,"15":0,"5":0}}}}}}