Logstash count key words in Kafka and increment statsd


(Chengcheng Pei) #1

New to logstash.
Assume messages in kafka look like:

{"key":"val1"}
{"key":"val2"}
{"key":"val2"}
{"key":"val1"}
...

Try to count the numbers of val1 and val2 in key field, and increase statsd keys: total.val1 and total.val2

How to write the filter and statsd output.
Searching by Google, hard to find good examples.

Any good example or suggestions?

input
{
    kafka
    {
        topic_id => 'test2'
    }
}


filter
{
    metrics
    {
        meter => ["'key':%{number}"]
        add_tag => ["metric"]
    }
}

output
{
    if "metric" in [tags]
    {
        stdout
        {
            codec => line
            {
                format => "count: %{number.count}"
            }
        }
        statsd { increment => "test.%{number.count}" }
    }
}

(Suyog Rao) #2

Not sure why you need metrics filter here. You can just do:

statsd { increment => "%{key}" }

statds will increment the val1, val2 key etc.


(Chengcheng Pei) #3

I need to metrics filter to count. Is there problem in my filter (meter) ?


(system) #4