Help with a ruby filter

[dmccuk (Dennis)]

Hi,

I'm getting the following message in logstash while sending metricbeat data to elasticsearch:

Sep 26 12:00:04 eurvlii16592 start.sh[58299]: [2018-09-26T12:00:04,355][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"logstash-metricbeat-daily-2018.09.26.11", :_type=>"doc", :_routing=>nil}, #LogStash::Event:0x22ee7a70], :response=>{"index"=>{"_index"=>"logstash-metricbeat-daily-2018.09.26.11", "_type"=>"doc", "_id"=>"cIOJFWYBkuJxonJVL5f4", "status"=>400, "error"=>{"type"=>"illegal_argument_exception", "reason"=>"mapper [system.diskio.iostat.write.per_sec.bytes] cannot be changed from type [long] to [float]"}}}}

If the value is categories in ES as a long value, I'm not able to use it in my graphs. I have a work around by adding a mutate filter into logstash:

filter {
mutate {
convert => { "[system][process][cpu][total][norm][pct]" => "float" }
convert => { "[system][diskio][iostat][request][avg_size]" => "float" }
convert => { "[system][process][memory][rss][pct]" => "float" }
convert => { "[system][process][cpu][total][pct]" => "float" }
convert => { "[system][diskio][iostat][queue][avg_size]" => "float" }
convert => { "[system][core][steal][pct]" => "float" }
convert => { "[system][cpu][steal][pct]" => "float" }
convert => { "[system][diskio][iostat][service_time]" => "float" }
convert => { "[system][core][softirq][pct]" => "float" }
convert => { "[system][diskio][iostat][busy]" => "float" }
convert => { "[system][diskio][iostat][write][request][per_sec]" => "float" }
convert => { "[system][diskio][iostat][write][request][merges_per_sec]" => "float" }
convert => { "[system][core][iowait][pct]" => "float" }
convert => { "[system][filesystem][used][pct]" => "float" }
convert => { "[system][core][system][pct]" => "float" }
convert => { "[system][diskio][iostat][write][per_sec.bytes]" => "float" }
convert => { "[system][memory][swap][used][pct]" => "float" }
}
}

But this list is only growing. I'd like to use a ruby filter to catch any/all long values and convert them to a "float" value instead of writing a large list like this.

Does anyone have any ruby pointers or examples I could use to achieve this or even a better way resolve this?

Thanks for your time.

Dennis

[Christian_Dahlqvist]

This conversion will change how the data is formatted in the JSON document sent to Elasticsearch, but not how Elasticsearch dynamically maps this. You need to solve this through the use of an index template in Elasticsearch where you map numeric values as floats.

[dmccuk (Dennis)]

Thanks for your reply @Christian_Dahlqvist,

I'm not too familiar with using templates but does this mean each and every metric needs to be listed out as a float in the template?

Something like this:

Translating the mutates:

[system][process][cpu][total][norm][pct]
[system][process][cpu][total][pct]
[system][process][memory][rss][pct]

become:

{
  "index_patterns": ["te*", "bar*"],
  "settings": {
    "number_of_shards": 1
  },
  "mappings": {
    "_doc": {
      "_source": {
        "enabled": false
      },
      "properties": {
        "system": {
          "process": {
            "cpu": {
              "total": {
                "norm": {
                  "pct": {
                   "type": "float"
                    }
                   },
                 "pct": {
                   "type": "float"
                    }
                 }
               }
              },
             "memory": {
              "rss": {
                "pct": {
                  "type": "float"
                   }
                }          
             }
          }
       }
    }
  }
}

[dmccuk (Dennis)]

That didn't help me much so I've stuck with the mutate for now.

I'm testing a ruby filter that takes any long (integer) number and converts it to float. That way everything arrives as the correct type in ES and everything i need works.

Anyone else get issues similar to this?