Metrics Filter - Trying to count and graph multiple event sources

Elastic Stack Logstash
1

What on earth I'm trying to do

I'm receiving events from lots of remote hosts, I want to produce metrics per event source. I'm pretty sure my method is flawed and looking for some help to get my on the right path for using elasticsearch as a metrics timeseries data store.

Over all metrics for each logstash receiver instance are super simple when counting overall message throughput. So I can get total throughput per logstash instance that are receiving remote events. Awesome :facepunch:

As the logstash instances are coming from a relatively large number of undetermined hosts, I am wanting to produce metrics per event source and tried to do so with this method:

Configuration

filter {
  if [message] {
    metrics {
      meter => [ "%{host}" ]
      add_tag => "metric"
    }
  }
}
output {
  # only emit events with the 'metric' tag
  if "metric" in [tags] {
    stdout {
      codec => rubydebug
    }
  }
}

Example output

{
"@version" => "1",
"@timestamp" => "2016-03-04T04:47:13.099Z",
"message" => "server1",
"server2" => {
        "count" => 37715,
        "rate_1m" => 91.41176979019558,
        "rate_5m" => 99.83427149846905,
        "rate_15m" => 120.76807446547991
},
"server3" => {
        "count" => 6095,
        "rate_1m" => 16.852501200722337,
        "rate_5m" => 22.17027922650242,
        "rate_15m" => 29.29614554210469
},
"server4" => {
        "count" => 61884,
        "rate_1m" => 130.19373481122938,
        "rate_5m" => 158.2217890726666,
        "rate_15m" => 193.45923174143326
},
"tags" => [
        [0] "metric"
]
}

As I am unable to know exactly which hosts will exist, I'm unsure how to compared the rate_1m nested fields beneath a variable number of objects that contain the host name of the event source server.

Ideally, I'd like to to split these up into separate events that populate fields with the same name but contain the host name of the original server.

It's almost like a I need a for loop to iterate through a state table of know source servers to produce the metrics or to split the above message into separate events.

I have managed to get it producing metrics to single parent object, however, I haven't determined a way with logstash to split at that object the lower objects into new messages:

"metrics" = {
        "server2" => {
                "count" => 37715,
                "rate_1m" => 91.41176979019558,
                "rate_5m" => 99.83427149846905,
                "rate_15m" => 120.76807446547991
        },
        "server3" => {
                "count" => 6095,
                "rate_1m" => 16.852501200722337,
                "rate_5m" => 22.17027922650242,
                "rate_15m" => 29.29614554210469
        },
        "server4" => {
                "count" => 61884,
                "rate_1m" => 130.19373481122938,
                "rate_5m" => 158.2217890726666,
                "rate_15m" => 193.45923174143326
        }
}

Ideally, from the data above, I'd like to line graph in Kibana or Grafana the rate_1m nested fields with the parent object source server name to distinguish the lines.

Is there a simple way to accomplish this?

2

Seems like there is a similar request for something along these lines which I didn't previously see in the Github page for the metrics filter plugin.

Generate one event per metric name

Guess I need to wait for the feature in the metrics filter for it to produce arbitrary metrics events based on variable field contents.