Using metric with dynamic field names

[kbop543 (Kyra S)]

Hey,
So I have been testing how to use metrics to tell me the count of how many times a logfile logged errors.

I use indexes to distinguish between logfiles by grabbing the filename from the path in a field called filename.
I have tried this in my logstash config file and it works:

metrics {
meter => "events"
add_tag => "metric"
}

output {
elasticsearch {
hosts => "g-elasticsearch:9200"
manage_template => false
index => "%{filename}"

}
stdout {codec => rubydebug}

if "metric" in [tags] and [events][count] > 1{
stdout {
codec => line {
format => "rate: %{[events][count]}"
}
}
}
}

This will give me the total count of all logs with errors. But I want it per logfile. I have tried various variations of this:

metrics {
meter => "%{filename}"
add_tag => "metric"
}

output {
elasticsearch {
hosts => "g-elasticsearch:9200"
manage_template => false
index => "%{filename}"

}
stdout {codec => rubydebug}

if "metric" in [tags] and [%{filename}][count] > 1{
stdout {
codec => line {
format => "rate: %{[%{filename}][count]}"
}
}
}
}

this does not work. In /var/log/logstash/.. it says "[2017-06-01T20:48:14,929][FATAL][logstash.runner ] An unexpected error occurred! {:error=>#<NoMethodError: undefined method >' for nil:NilClass>, :backtrace=>["(eval):175:inoutput_func'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:400:in output_batch'", "org/jruby/RubyProc.java:281:incall'", "/usr/share/logstash/logstash-core/lib/logstash/util/wrapped_synchronous_queue.rb:227:in each'", "org/jruby/RubyHash.java:1342:ineach'", "/usr/share/logstash/logstash-core/lib/logstash/util/wrapped_synchronous_queue.rb:226:in each'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:394:inoutput_batch'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:352:in worker_loop'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:317:instart_workers'"]}"

I know that the syntax is somehow wrong. What is the right syntax? I have tried variations of the above to see which one would work, aka switching around curly braces or quotation marks but nothing seems to work!

Hoping someone can help me,
Thanks

[pts0]

Hi,

I suppose the problem is in the output part (first of all just format it nice in the post ):

output {
    elasticsearch {
              hosts => "g-elasticsearch:9200"
              manage_template => false
              index => "%{filename}"
    }
    stdout {codec => rubydebug}

    if "metric" in [tags] and [%{filename}][count] > 1{
         stdout {
               codec => line {
                    format => "rate: %{[%{filename}][count]}"
               }
         }
    } 
}

I m not a compiler, but I think the problem is if "metric" in [tags] and [%{filename}][count] > 1

Have a look at https://www.elastic.co/guide/en/logstash/current/event-dependent-configuration.html and make sure your if expression is valid.
To test it try empthy if or something trivial in it, , maybe is format => "rate: %{[%{filename}][count]}" the problem.

pts0

[kbop543 (Kyra S)]

The expression is valid because it works totally fine with [events][count] > 1.

The only different between the following
[events][count] > 1
[%{filename}][count] > 1

..is that I added a %{filename} instead of a regular 'events'. Not sure why it won't work. I am assuming there is a syntax to accommodate this dynamic field but it's not documented anywhere...