Hi,
I'm getting the following message in logstash while sending metricbeat data to elasticsearch:
Sep 26 12:00:04 eurvlii16592 start.sh[58299]: [2018-09-26T12:00:04,355][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"logstash-metricbeat-daily-2018.09.26.11", :_type=>"doc", :_routing=>nil}, #LogStash::Event:0x22ee7a70], :response=>{"index"=>{"_index"=>"logstash-metricbeat-daily-2018.09.26.11", "_type"=>"doc", "_id"=>"cIOJFWYBkuJxonJVL5f4", "status"=>400, "error"=>{"type"=>"illegal_argument_exception", "reason"=>"mapper [system.diskio.iostat.write.per_sec.bytes] cannot be changed from type [long] to [float]"}}}}
If the value is categories in ES as a long value, I'm not able to use it in my graphs. I have a work around by adding a mutate filter into logstash:
filter {
mutate {
convert => { "[system][process][cpu][total][norm][pct]" => "float" }
convert => { "[system][diskio][iostat][request][avg_size]" => "float" }
convert => { "[system][process][memory][rss][pct]" => "float" }
convert => { "[system][process][cpu][total][pct]" => "float" }
convert => { "[system][diskio][iostat][queue][avg_size]" => "float" }
convert => { "[system][core][steal][pct]" => "float" }
convert => { "[system][cpu][steal][pct]" => "float" }
convert => { "[system][diskio][iostat][service_time]" => "float" }
convert => { "[system][core][softirq][pct]" => "float" }
convert => { "[system][diskio][iostat][busy]" => "float" }
convert => { "[system][diskio][iostat][write][request][per_sec]" => "float" }
convert => { "[system][diskio][iostat][write][request][merges_per_sec]" => "float" }
convert => { "[system][core][iowait][pct]" => "float" }
convert => { "[system][filesystem][used][pct]" => "float" }
convert => { "[system][core][system][pct]" => "float" }
convert => { "[system][diskio][iostat][write][per_sec.bytes]" => "float" }
convert => { "[system][memory][swap][used][pct]" => "float" }
}
}
But this list is only growing. I'd like to use a ruby filter to catch any/all long values and convert them to a "float" value instead of writing a large list like this.
Does anyone have any ruby pointers or examples I could use to achieve this or even a better way resolve this?
Thanks for your time.
Dennis