Hi @paz,
Thanks for the idea, actually, it turns out that's exactly what i'm attempting to move away from and have been using this method for a couple of years. I started with statsd, but now telegraf has a statsd compatible input and ingest that data into influxdb.
The problem is these increment packets is that they are UDP, and increment relies on the counters being incremented reliably, which it's fundamentally not. The eps rates of the metric filter would be far less spammy and means I could send them using TCP as gauge metrics.
It's possible I could send the metrics events output to a custom golang service to munge it, but it seems like a function that either the logstash-filter-split or logstash-filter-metrics plugin could of been capable of.
I've had an idea of using a separate elasticsearch cluster for the the log cluster metrics for some time, however, this metrics splitting is one of the problems for moving away from statsd and inflxudb.