Hi,
I wanted to see how rate at which the clients were sending logs to logstash. I wanted to use the metric filter plugin. And it is working. Except one thing. Here is the conf:
input
{
http
{
port => 9999
codec => json
id => "demo"
type => "counted"
tags => "http"
}
}
filter
{
split
{
field => "events"
}
metrics
{
meter => "events"
add_tag => "metric"
add_field => { "hostname" => "%{host}"}
}
}
output
{
if "metric" in [tags]
{
stdout
{
codec => line
{
format => "host:%{hostname}, count_9999: %{[events][count]}, rate_1m: %{[events][rate_1m]}"
}
}
}
else
{
elasticsearch
{
hosts => "http://starkindustries:9455"
ilm_enabled => "true"
ilm_rollover_alias => "test"
ilm_pattern => "000001"
ilm_policy => "test_policy"
user => 'tony'
password => 'pepper'
}
}
}
I am getting output like: host:%{host}, count_9999: 17, rate_1m: 3.128151
then how will my format string look like? format => "host:%{hostname}, count_9999: %{[events][count]}, rate_1m: %{[events][rate_1m]}" did not work
and format => "host:%{hostname}, count_9999: %{[events_%{host}][count]}, rate_1m: %{[events_%{host}][rate_1m]}"
gave me compilation errors.
I think I am missing some formatting here.
For a meter => "thing" you will receive the following fields:
"[thing][count]" - the total count of events
etc. If you use meter => "events_%{host}" for an event which has the [host] field set to "foo" you will get [events_foo][count] etc. If the event has [host] "bar" you will get [events_bar][count] etc. metrics
I am a bit lost here. Does this mean that I will have to hard code the expected event name in the format string? If so that will not solve the issue for me.
I will do some experiments and come back.
You might want to explain why you are using that format string. Why not just use json_lines?
If you absolutely must have a [host] field then you could do something like this in your filter section after the metrics filter.
if "metric" in [tags]
ruby {
code => '
event.to_hash.each { |k, v|
matchdata = k.match(/events_([\w\.]+)/)
if matchdata
event.set("events", v)
event.set("host", matchdata[0])
event.remove(k) # Not actually needed if your format does not reference [events_foo]
end
}
'
}
}
I have not tested that, but it shows the general idea.
If [\w\.]+ is too simple to match your host names you could get fancier, all the way up to the pattern grok uses
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.