Top hits for recent values not working for numbers that give floating point results

[keiranmraine (Keiran Raine)]

I'm using metricbeat to gather system information. I can clearly see the values are being populated in the discover pane when filtering and splitting as required for the host, cpu_id to ensure there are no nulls from unrelated entries

Applying this same logic to the visualisation data table I find that system.idle.pct always shows as 0 when I use the 'Top Hits' aggregation to try and get the latest value (Max, Size=1). Of note is that if I try to use, avg, concat I get - instead of any value.

I've included the payload of the request at the end.

I'm looking at this to try and have a "current" CPU stat on a gauge, this became evident while debugging why this wasn't working.

I'm using 6.5.4 of the stack.

Please note that the response shows the fraction values so it appears to be something in the UI layer

Payload
{"index":"metricbeat*","ignore_unavailable":true,"preference":1547814588936}

{"aggs":{"2":{"terms":{"field":"host.name.keyword","size":5,"order":{"_key":"desc"}},"aggs":{"3":{"terms":{"field":"system.core.id","size":5,"order":{"_key":"asc"}},"aggs":{"4":{"top_hits":{"docvalue_fields":[{"field":"system.core.idle.pct","format":"use_field_mapping"}],"_source":"system.core.idle.pct","size":1,"sort":[{"@timestamp":{"order":"desc"}}]}}}}}}},"size":0,"_source":{"excludes":[]},"stored_fields":["*"],"script_fields":{},"docvalue_fields":[{"field":"@timestamp","format":"date_time"}],"query":{"bool":{"must":[{"range":{"@timestamp":{"gte":1547816442402,"lte":1547816742402,"format":"epoch_millis"}}},{"match_phrase":{"metricset.module.keyword":{"query":"system"}}},{"match_phrase":{"metricset.name.keyword":{"query":"core"}}}],"filter":[{"match_all":{}}],"should":[],"must_not":[]}},"timeout":"30000ms"}

Response

{"responses":[{"took":9,"timed_out":false,"_shards":{"total":5,"successful":5,"skipped":0,"failed":0},"hits":{"total":120,"max_score":0.0,"hits":[]},"aggregations":{"2":{"doc_count_error_upper_bound":0,"sum_other_doc_count":0,"buckets":[{"key":"c0b7d1eb8c68","doc_count":60,"3":{"doc_count_error_upper_bound":0,"sum_other_doc_count":0,"buckets":[{"key":0,"doc_count":30,"4":{"hits":{"total":30,"max_score":null,"hits":[{"_index":"metricbeat-6.4.2-2019.01.18","_type":"doc","_id":"OyMRYWgBk-_510orDXse","_score":null,"_source":{"system":{"core":{"idle":{"pct":0.9608}}}},"fields":{"system.core.idle.pct":[0]},"sort":[1547816732876]}]}}},{"key":1,"doc_count":30,"4":{"hits":{"total":30,"max_score":null,"hits":[{"_index":"metricbeat-6.4.2-2019.01.18","_type":"doc","_id":"PCMRYWgBk-_510orDXse","_score":null,"_source":{"system":{"core":{"idle":{"pct":0.9588}}}},"fields":{"system.core.idle.pct":[0]},"sort":[1547816732876]}]}}}]}},{"key":"093107f9d7ee","doc_count":60,"3":{"doc_count_error_upper_bound":0,"sum_other_doc_count":0,"buckets":[{"key":0,"doc_count":30,"4":{"hits":{"total":30,"max_score":null,"hits":[{"_index":"metricbeat-6.4.2-2019.01.18","_type":"doc","_id":"QiMRYWgBk-_510orDntU","_score":null,"_source":{"system":{"core":{"idle":{"pct":0.99}}}},"fields":{"system.core.idle.pct":[0]},"sort":[1547816733178]}]}}},{"key":1,"doc_count":30,"4":{"hits":{"total":30,"max_score":null,"hits":[{"_index":"metricbeat-6.4.2-2019.01.18","_type":"doc","_id":"PSMRYWgBk-_510orDntT","_score":null,"_source":{"system":{"core":{"idle":{"pct":0.992}}}},"fields":{"system.core.idle.pct":[0]},"sort":[1547816733178]}]}}}]}}]}},"status":200}]}

[cufflinks (John Dorlus)]

Asking around the kibana team to see if there is some additional config that can help with this.

[LeeDr (Lee Drengenberg)]

I can reproduce this if I don't filter out docs which don't have the field.

I created a Saved Search named metricbeat idle pct

And then created a Data Table visualization using that Saved Search. Now it does show Average instead of -;

And Top Hits;

[keiranmraine (Keiran Raine)]

Creating a search for exists instead of filtering down to the field does appear to solve the problem. Thanks.