Hey Elastic Folks:
I'm using metricbeat to collect cpu utilization data from a number of linux hosts, publishing directly to elasticsearch. I'm seeing cases where pct type fields are getting mapped in elasticsearch as long's instead of their expected floats if the first document published has 0 for that pct field.
It appears that when one of the metricbeat clients has to create a new index (day change), and it sees the a pct field having a value of 0 (in my case, system.cpu.total.norm.pct), metricbeat creates the new index with a mapping type of long for that field and saves 0 for the fields value (not 0.0). If the first metricbeat of the day is publishing a value greater than 0, which is a majority of the time, it properly maps the field as a float.
I'm using a custom index naming pattern, and have loaded a template into elasticsearch via metricbeat setup. In order to workaround I've been manually creating a new index, setting the mapping to use float for the pct field I care about, and re-indexing from old (long) to new (float).
Here's some info on my setup:
Deployment
- Elasticsearch 6.2.0, via docker.elastic.co/elasticsearch/elasticsearch:6.2.0 - issue also seen on 6.1.3
- Metricbeat 6.1.3 and 6.2.0
metricbeat.yml
output.elasticsearch:
enabled: true
hosts: ["elasticsearch1.domain.com", "elasticsearch2.domain.com"]
index: "custom-index-%{+yyyy.MM.dd}"
setup.template:
name: "custom-index"
pattern: "custom-index"
metricbeat.modules:
- module: system
period: 1m
metricsets:
- cpu
cpu.metrics: [normalized_percentages]
processors:
- include_fields:
fields: ["system.cpu.total.norm.pct", "fields", "beat", "metricset"]
name: some-custom-name.domain.com
fields:
... a number of custom fields, all strings, all mapping correctly 100% of the time
elasticsearch template for my specific field having issues
... under system.cpu
"total" : {
"properties" : {
"pct" : {
"scaling_factor" : 1000,
"type" : "scaled_float"
},
"norm" : {
"properties" : {
"pct" : {
"type" : "scaled_float",
"scaling_factor" : 1000
}
}
},
"value" : {
"type" : "long"
}
}
}, ...
broken mapping for an index who's first document had a zero (0) for that field
... under system.cpu
"total" : {
"properties" : {
"norm" : {
"properties" : {
"pct" : {
"type" : "long" ### This should be a float, and is a float if the first document has a value non-zero
}
}
}
}
}, ...