Properly Setup Metricbeat with Statsd

Elastic Stack Beats
1

I'm using Metricbeat to send to a 3rd party logstash, and it's working well with system but there seems to be some trouble with statsd (running on localhost:8125). My current metricbeat.yml looks like this (with some other logstash params):

metricbeat.config.modules:
  # Glob pattern for configuration loading
  path: ${path.config}/modules.d/*.yml

  # Set to true to enable config reloading
  reload.enabled: false


metricbeat.modules:
- module: statsd
  host: localhost
  port: 8125
  metricsets: 
    - server
  enabled: true

# ======================= Elasticsearch template setting =======================

setup.template.settings:
  index.number_of_shards: 1
  index.codec: best_compression
  #_source.enabled: false

Now, I do ./metricbeat modules enable statsd and then ./metricbeat -e to watch it run. I get this:

2020-10-12T11:59:43.447-0400	DEBUG	[module]	module/wrapper.go:181	statsd/server will start after 3.081425219s
2020-10-12T11:59:43.447-0400	DEBUG	[cfgfile]	cfgfile/cfgfile.go:193	Load config from file: /Users/xxx/code/metricbeat-7.9.2-darwin-x86_64/modules.d/statsd.yml
2020-10-12T11:59:43.448-0400	DEBUG	[cfgfile]	cfgfile/reload.go:146	Number of module configs found: 1
2020-10-12T11:59:43.449-0400	INFO	cfgfile/reload.go:164	Config reloader started
2020-10-12T11:59:43.449-0400	DEBUG	[cfgfile]	cfgfile/reload.go:194	Scan for new config files
2020-10-12T11:59:43.449-0400	DEBUG	[cfgfile]	cfgfile/cfgfile.go:193	Load config from file: /Users/xxx/code/metricbeat-7.9.2-darwin-x86_64/modules.d/statsd.yml

But as soon as I view /Users/xxx/code/metricbeat-7.9.2-darwin-x86_64/modules.d/statsd.yml, it says its disabled!:

- module: statsd
  host: "localhost"
  port: "8125"
  enabled: false

I finally edited the statsd.yml, setup again, and ran it again, but still to no avail. No metrics are coming in from statsd (though statsd is most definitely tracking them). For what it's worth, I also did get this one error (though it seems to have disappeared now):

2020-10-12T11:43:08.892-0400	ERROR	runtime/panic.go:212	recovered from panic while fetching 'statsd/server' for host ''. Recovering, but please report this.	

{"panic": "runtime error: invalid memory address or nil pointer dereference", "stack": "github.com/elastic/beats/v7/libbeat/logp.Recover\n\t/go/src/github.com/elastic/beats/libbeat/logp/global.go:101\nruntime.gopanic\n\t/usr/local/go/src/runtime/panic.go:969\nruntime.panicmem\n\t/usr/local/go/src/runtime/panic.go:212\nruntime.sigpanic\n\t/usr/local/go/src/runtime/signal_unix.go:695\ngithub.com/elastic/beats/v7/metricbeat/helper/server/udp.(*UdpServer).Stop\n\t/go/src/github.com/elastic/beats/metricbeat/helper/server/udp/udp.go:122\ngithub.com/elastic/beats/v7/x-pack/metricbeat/module/statsd/server.(*MetricSet).Run\n\t/go/src/github.com/elastic/beats/x-pack/metricbeat/module/statsd/server/server.go:102\ngithub.com/elastic/beats/v7/metricbeat/mb/module.(*metricSetWrapper).run\n\t/go/src/github.com/elastic/beats/metricbeat/mb/module/wrapper.go:203\ngithub.com/elastic/beats/v7/metricbeat/mb/module.(*Wrapper).Start.func1\n\t/go/src/github.com/elastic/beats/metricbeat/mb/module/wrapper.go:147"}

I have a few questions on Metricbeat with statsd:

  • what is the difference between modules.d/statsd.yml and metricbeat.modules in metricbeat.yml, and do I have to set both up?
  • is there anything on the statsd side that I have to do to make it compatible with Metricbeat, beyond matching the IP and port?
  • are there any simple examples of Metricbeat actually working with statsd? I can't seem to find a single one online
2
  1. You're good to go with setting up only files in modules.d.
  2. I don't thing so, it's not a complex integration. Did you try to enable the debug mode?
  3. This is relatively simple integration, there is no more configuration than: Statsd module | Metricbeat Reference [8.11] | Elastic

For the panic, I opened a Beats issue: [Metricbeat][statsd] runtime error: invalid memory address or nil pointer dereference · Issue #21733 · elastic/beats · GitHub

3

Awesome thats very helpful, answers all of my questions.

Thanks for creating the issue, it keeps surfacing so I have a feeling thats whats making this not work.

4

For what it's worth, I just got it working (I didn't realize Metricbeat is already running a Statsd service). The issue was just that I was defining the statsd config twice, once in metricbeat.config.modules and once in metricbeat.modules. I deleted metricbeat.config.modules and its working perfectly now.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.