Not able to add host name to metric filter plugin output

Elastic Stack Logstash
#1

Hi,
I wanted to see how rate at which the clients were sending logs to logstash. I wanted to use the metric filter plugin. And it is working. Except one thing. Here is the conf:

input
{
        http
        {
                port => 9999
                codec => json
                id => "demo"
                type => "counted"
                tags => "http"
        }
}

filter
{   
        split
        {
                field => "events"
        }

        metrics
        {
                meter => "events"
                add_tag => "metric"
                add_field => { "hostname" => "%{host}"}
        }
}

output
{
        if "metric" in [tags]
        {
                stdout
                {
                        codec => line
                        {
                                format => "host:%{hostname}, count_9999: %{[events][count]}, rate_1m: %{[events][rate_1m]}"
                        }
                }
        }
        else
        {
                elasticsearch
                {
                        hosts => "http://starkindustries:9455"
                        ilm_enabled => "true"
                        ilm_rollover_alias => "test"
                        ilm_pattern => "000001"
                        ilm_policy => "test_policy"
                        user => 'tony'
                        password => 'pepper'
                }
        }
}

I am getting output like:
host:%{host}, count_9999: 17, rate_1m: 3.128151

I think I am missing something here.

#2

The metrics filter generates a new event. It does not have any fields that can be sprintf'd. This is a known issue.

#3

Do we have any workarounds? It will be super useful as I see some of the client machines lagging compared to others.

#4

If you want per-host meters then use meter => "events_%{host}"

#5

If I use
meter => "events_%{host}"

then how will my format string look like?
format => "host:%{hostname}, count_9999: %{[events][count]}, rate_1m: %{[events][rate_1m]}" did not work
and
format => "host:%{hostname}, count_9999: %{[events_%{host}][count]}, rate_1m: %{[events_%{host}][rate_1m]}"
gave me compilation errors.
I think I am missing some formatting here.

#6

The documentation covers this.

For a meter => "thing" you will receive the following fields:

  • "[thing][count]" - the total count of events

etc. If you use meter => "events_%{host}" for an event which has the [host] field set to "foo" you will get [events_foo][count] etc. If the event has [host] "bar" you will get [events_bar][count] etc. metrics

#7

I am a bit lost here. Does this mean that I will have to hard code the expected event name in the format string? If so that will not solve the issue for me.
I will do some experiments and come back.

#8

You might want to explain why you are using that format string. Why not just use json_lines?

If you absolutely must have a [host] field then you could do something like this in your filter section after the metrics filter.

if "metric" in [tags]
    ruby {
        code => '
            event.to_hash.each { |k, v|
                matchdata =  k.match(/events_([\w\.]+)/)
                if matchdata
                    event.set("events", v)
                    event.set("host", matchdata[0])
                    event.remove(k) # Not actually needed if your format does not reference [events_foo]
                end
            }
        '
    }
}

I have not tested that, but it shows the general idea.

If [\w\.]+ is too simple to match your host names you could get fancier, all the way up to the pattern grok uses

\b(?:[0-9A-Za-z][0-9A-Za-z-]{0,62})(?:\.(?:[0-9A-Za-z][0-9A-Za-z-]{0,62}))*(\.?|\b)
#9

Thanks a lot for this. This solves my problem. :grinning: