How does other field work in Kibana's Piechart

[gevmart (Gevorg)]

Hey there,

I was playing around with Kibana 6.3 piecharts, and exploring the other field of the piechart. I was exploring the following situation.

Suppose the documents stored in kibana have a text type value name, and numeric value assets. I do a term aggregation with respect to name, and display the sum of the assets on the piechart. Suppose the number of the names is bigger than the specified size of the aggregation and other option is enabled. Kibana correctly displays all the aggregations, and other field is also computed correctly.

The question is that when I looked at the response Kibana gets, it looks like this

{
"took": 2,
"timed_out": false,
"_shards": {
"total": 5,
"successful": 5,
"skipped": 0,
"failed": 0
},
"hits": {
"total": 30,
"max_score": 0,
"hits":
},
"aggregations": {
"2": {
"doc_count_error_upper_bound": 0,
"sum_other_doc_count": 9,
"buckets": [
{
"1": {
"value": 34
},
"key": "a",
"doc_count": 8
},
{
"1": {
"value": 29
},
"key": "c",
"doc_count": 7
},
{
"1": {
"value": 27
},
"key": "b",
"doc_count": 6
}
]
}
},
"status": 200
}

It has all the values corresponding to the first three keys (size of the aggregation is three), but does not have any value for the other field. So I was wondering how does Kibana know what is the value of the other field. (I am mostly curious, but if there is an easy solution it would be useful for the project I am working on).

Thank you

[bhavyarm]

Hello,

If you see the top of your response - you can see the other value response which Kibana gets.
Which in your case is 9.

@gevmart

"sum_other_doc_count": 9,

This is our es document on other filter: Filters Aggregation | Elasticsearch Guide [6.3] | Elastic

Hope that helps.

Thanks,
Bhavya

[gevmart (Gevorg)]

Hi,

Thanks for the reply.

sum_other_doc_count is the number of documents that have not been included in the aggregation (which is 9 in this case). The piechart, however, shows the sum of grades which correspond to the documents not included, which is 39 in this case (as can be seen on the picture). How does Kibana get this value?

[bhavyarm]

argh ..hold on I totally missed that.

[bhavyarm]

Can you please post screenshots of your editor on the left side? For me in my local: what I see for sum_other_doc_count matches what I see on the screen.

So - not sure whats happening with this?
Also whats the version of the stack you are on?

Thanks,
Bhavya

[bhavyarm]

This is the pull request which got this change in to Kibana:

other and missing bucket support for terms agg

Thanks,
Bhavya

[gevmart (Gevorg)]

I guess the difference is the metric we are using. I am using a sum metric

If the metric was count, sum_other_docs_count could be used to calculate other field, but I don't how it works when other metric is used (like sum in this case).

I am using Elastic Stack 6.3

Gevorg

[bhavyarm]

@timroes or @ppisljar can I please have your input when you have 5 mins here?

Thank you!
Bhavya

[timroes]

Hi Gevorg,

Other buckets work by doing two requests against Elasticsearch. One which will actually retrieve the terms aggregation and it's values as if you would not have specified other terms. After that has returned, we are doing a second request, which is basically a request using the same metric aggregations, but is using a filter, to filter out all of the terms that has been returned by the first request in the terms aggregation.

In case you've got multiple nested terms aggregations, we will include a filter aggregation for every level in the second request.

What metric you actually choose doesn't matter. It works the same with Count as with any other metric, since we do the second request with a different filters bucket aggregation, but can use the very same metrics aggregations.

With Kibana 6.4, we are replacing the existing spy panel, by a new solution, which will also be able to show you both requests (the data request and the other bucket request), so you can gain better insights into that.

Hope that answer contained all the information you where looking for. If not please feel free to get back with any questions.

Cheers,
Tim

[gevmart (Gevorg)]

Thanks Tim. Your reply answers everything that interested me. It all makes perfect sense.

Best,
Gevorg