Elasticsearch aggregation shows 0 instead of actual value

Elastic Stack Elasticsearch
1

[EDIT] - Moved thread from Kibana to Elasticsearch following findings in this post.

Hello,

I'm having trouble displaying some data properly in Kibana. I have a field called "loss", which ranges from 0.00 to 0.05 in my current dataset. I formatted it as a percentage in Kibana (0% to 5%) and it displays properly in search results:

But when setting up a Visualization, "loss" will always show 0%:

I am currently using "Max" aggregation on a line chart with a time interval of one second, but I have tried many different chart types (including Data Table), aggregation modes (average), time intervals and chart settings (Y-Axis scaling). I also tried to display this field without formatting (raw decimal values) but nothing seems to do the trick.

Other fields from the same dataset display properly, and if I create a scripted field with a static value in the same range (0.01 / 1%) it will be displayed properly on the same visualization. Is there something that I missed?

Thanks in advance

2

What version of Kibana are you using? I tried using some Metricbeat data with values in the same interval, marked as percentages and I didn't see any troubles in displaying them.
Could you save the visualization and attach the exported object here? (or paste the JSON).
Also, under each visualization there's an "Arrow up" that opens a panel with the request and response from ES. Can you also paste the request here? Response would be nice as well, but if you have any sensitive data, you could have a look at it an see if anything looks weird there.

3

I'm using version 5.2.2. I also pasted response for one point of data as an example, but I can paste more, almost all data is dummy.

Visualization export:

[
{
    "_id": "a5561cc0-0ed0-11e7-bb36-cfca88d9d1a6",
    "_type": "visualization",
    "_source": {
        "title": "packet_loss",
        "visState": "{\"title\":\"packet_loss\",\"type\":\"line\",\"params\":{\"addLegend\":true,\"addTimeMarker\":false,\"addTooltip\":true,\"defaultYExtents\":false,\"drawLinesBetweenPoints\":true,\"interpolate\":\"linear\",\"legendPosition\":\"bottom\",\"radiusRatio\":9,\"scale\":\"linear\",\"setYExtents\":false,\"showCircles\":false,\"times\":[]},\"aggs\":[{\"id\":\"1\",\"enabled\":true,\"type\":\"max\",\"schema\":\"metric\",\"params\":{\"field\":\"loss\"}},{\"id\":\"2\",\"enabled\":true,\"type\":\"date_histogram\",\"schema\":\"segment\",\"params\":{\"field\":\"stamp_inserted\",\"interval\":\"auto\",\"customInterval\":\"2h\",\"min_doc_count\":1,\"extended_bounds\":{}}},{\"id\":\"3\",\"enabled\":true,\"type\":\"max\",\"schema\":\"metric\",\"params\":{\"field\":\"treshold_loss\",\"customLabel\":\"Treshold\"}}],\"listeners\":{}}",
        "uiStateJSON": "{\"vis\":{\"colors\":{\"Max loss\":\"#BF1B00\",\"Average treshold_loss\":\"#58140C\",\"Treshold\":\"#58140C\"}}}",
        "description": "",
        "savedSearchId": "7ba014f0-0ece-11e7-8cf4-2f3c2fc6a19e",
        "version": 1,
        "kibanaSavedObjectMeta": {
            "searchSourceJSON": "{\"filter\":[]}"
        }
    }
}
]

Request:

{
"size": 0,
"aggs": {
    "2": {
        "date_histogram": {
            "field": "stamp_inserted",
            "interval": "1s",
            "time_zone": "Europe/Berlin",
            "min_doc_count": 1
        },
        "aggs": {
            "1": {
                "max": {
                    "field": "loss"
                }
            },
            "3": {
                "max": {
                    "script": {
                        "inline": "0.01",
                        "lang": "painless"
                    }
                }
            }
        }
    }
},
"highlight": {
    "pre_tags": [
        "@kibana-highlighted-field@"
    ],
    "post_tags": [
        "@/kibana-highlighted-field@"
    ],
    "fields": {
        "*": {}
    },
    "require_field_match": false,
    "fragment_size": 2147483647
},
"query": {
    "bool": {
        "must": [
            {
                "query_string": {
                    "query": "peer_ip_src:[EDITED]",
                    "analyze_wildcard": true
                }
            },
            {
                "range": {
                    "stamp_inserted": {
                        "gte": 1490105400000,
                        "lte": 1490105700000,
                        "format": "epoch_millis"
                    }
                }
            }
        ],
        "must_not": []
    }
},
"_source": {
    "excludes": []
}
}

Response:

{
"took": 3,
"timed_out": false,
"_shards": {
    "total": 5,
    "successful": 5,
    "failed": 0
},
"hits": {
    "total": 1,
    "max_score": 0,
    "hits": []
},
"aggregations": {
    "2": {
        "buckets": [
            {
                "1": {
                    "value": 0
                },
                "3": {
                    "value": 0.01
                },
                "key_as_string": "2017-03-21T15:12:09.000+01:00",
                "key": 1490105529000,
                "doc_count": 1
            }
        ]
    }
},
"status": 200
}

Actual data from the Search:

{
"_index": "netflow",
"_type": "netflow_data",
"_id": "[EDITED]",
"_score": null,
"_source": {
    "iface_in": 11,
    "iface_out": 200,
    "peer_ip_src": "[EDITED]",
    "ip_src": "[EDITED]",
    "nbar_name": "rtp-audio",
    "class_dscp": 0,
    "ip_dst": "[EDITED]",
    "packets": 2358803,
    "loss": 0.0491,
    "jitter": 47711,
    "delay": 276,
    "@timestamp": "2017-03-22T07:07:36.304Z",
    "stamp_inserted": "2017-03-21T14:12:09.000Z",
    "flows": 5,
    "bytes": 1412937895,
    "@version": "1",
    "tos": 0
},
"fields": {
    "treshold_loss": [
        0.01
    ],
    "@timestamp": [
        1490166456304
    ],
    "treshold_jitter": [
        30
    ],
    "stamp_inserted": [
        1490105529000
    ],
    "treshold_delay": [
        150
    ]
},
"sort": [
    1490105529000
]
}
4

In this example, it looks like the max value for the loss field coming from ES is actually 0.

5

That is what I don't understand, the same record from search under "Discover" module shows 0.0491 for loss, see the last snippet.

"loss": 0.0491

Is there a difference in how the data is fetched?

6

Yeah, that's a conundrum. I'm not seeing any glaring issues. You might try playing around with the raw aggregation query in Console to see if you can figure out why that value is coming back as 0.

7

You were right, it is the aggregation.

image.png1423×830 84.6 KB

I'm not sure how to address this though, is there a way to debug/verbose the aggregation?

8

What datatype is the loss field mapped as?

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.