I've been using statsd, telegraf (when using InfluxDB) and datadog agent (when using datadog) with great results for metric collection, but now that I'm trying to use elastic search as the tsdb for my metrics I'm having a tough time finding the best agent to collect the metrics (through udp server preferrably, instead of log files) .
Big problem with is telegraf still doesn't support ES as a backend (request is open), and statsd agent doesn't support custom tags, making it a lot less flexible for visualizations like kibana or graphana.
I think it would be a great addition to metric beats modules, and enable easy shipping of metrics to ES.
I recently had a similar case, although I mostly wanted to log data from individual applications locally via UDP socket, and then ship them via the desired output. There was already a community beat (udpbeat) that accomplished some of what I need, so I decided to create another updated beat called udplogbeat.
Keep in mind I'm not familiar with the specifics of your use case and how high of a volume you would log, but check out that custom beat in case it works for you. Keep in mind if the remote connection for some reason fails and the beat can't connect to your indicated output, the data you log is not persisted to disk. Also, this project is fairly new, so there may be issues with it. If you notice anything, feel free to submit an issue.
One thing that is different about statsd compared to all the other modules in metricbeat is, that it would be push instead of pull. So instead of metricbeat fetching the data, some "collectors" which output statsd formatted data would push it metricbeat.
What are your systems that output statsd metrics? In some cases, it could be better to just have metricbeat fetch the metrics directly from these systems instead of having statsd in the middle.
Yes, I find that pushing metrics through UDP is more practical (though not always cleaner) than creating endpoints just for metrics. After working with statsd for some time I realized that it motivates developers to experiment and optimize much more often by using metrics.
You're correct, it's the closest to what I need, but I asked here because I was looking for a more mature project, like telegraf or statsd. Unfortunately udpbeat seems not to be maintained anymore, and doesn't support plugins and the likes.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.
