# Confusion Regarding Unstable Bucket-ing in Kibana Visualizations **URL:** \ **Category:** Kibana\ **Created:** [January 26, 2018, 4:56pm UTC](https://discuss.elastic.co/t/confusion-regarding-unstable-bucket-ing-in-kibana-visualizations/117237 "2018-01-26T16:56:03Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![frankottey](https://avatars.discourse-cdn.com/v4/letter/f/5fc32e/32.png) [@frankottey](https://discuss.elastic.co/u/frankottey)\ **Post date:** [January 26, 2018, 4:56pm UTC](https://discuss.elastic.co/t/confusion-regarding-unstable-bucket-ing-in-kibana-visualizations/117237/1 "2018-01-26T16:56:03Z")
Hello everyone, I have been currently exploring Kibana visualizations of some test data, specifically horizontal bar charts. I had been attempting to bucket the x-axis by aggregation of terms when I was surprised to notice that, when using count as the ordering metric, manipulating the depth of the descending ordering caused the actual values of the counts results to change! Specifically, I was looking at the top 10 fields in descending order, and their ordering was different depending on whether I had the order size set at 10 or 50. Furthermore, the total counts when hovering over the individual field results didn't always agree with the results of the count of a naive query on the field from the dataset in the "Discover" tab. Specifically, when I queried the specific index for the specific "top" field, the count that was returned matched the count returned when I windowed the visualization ordering size to 50, but not to the resulting count of the same field when the ordering size was windowed to 10; there, the count was a full 20 items less! Has anyone else experienced unstable sorting, or is there any explanation as to why changing the size of the ordering has an impact on the results returned? Regards, Frank ---
**Author:** ![LeeDr](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leedr/32/9289_2.png) [@LeeDr](https://discuss.elastic.co/u/LeeDr)\ **Post date:** [January 26, 2018, 8:12pm UTC](https://discuss.elastic.co/t/confusion-regarding-unstable-bucket-ing-in-kibana-visualizations/117237/2 "2018-01-26T20:12:57Z")
If this data is OK to share maybe you could post a couple of screenshots? Or redact the private info out of the screenshot with a paint program? There are some cases where Elasticsearch returns approximate results and the _could_ be what you're seeing (more likely on large clusters). [https://www.elastic.co/guide/en/elasticsearch/guide/current/\_approximate\_aggregations.html](https://www.elastic.co/guide/en/elasticsearch/guide/current/_approximate_aggregations.html) or [https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-metrics-cardinality-aggregation.html#\_counts\_are\_approximate](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-metrics-cardinality-aggregation.html#_counts_are_approximate) But there might be a different issue at play here. If you can post more details about your exact aggregations we can take a look at that. ---
**Author:** ![frankottey](https://avatars.discourse-cdn.com/v4/letter/f/5fc32e/32.png) [@frankottey](https://discuss.elastic.co/u/frankottey)\ **Post date:** [January 29, 2018, 2:54pm UTC](https://discuss.elastic.co/t/confusion-regarding-unstable-bucket-ing-in-kibana-visualizations/117237/3 "2018-01-29T14:54:23Z")
> [@LeeDr](#): > > be you could post a couple of screenshots? > > Or redact the private info out of the screenshot with a paint program? > > There are some cases where Elasticsearch returns approximate results and the could be what you're seeing (more likely on large clusters). Hi Lee, Thank you for reaching out. I have only installed Elasticsearch and Kibana on a single machine, so I don't think that I'm running into any eventual consistency or accuracy issues related to Elasticsearch's distributed architecture, but I could be wrong. Another factor here is that the data provided from all these logs is a couple weeks in the past and is no longer changing. Here's some screen shots: ![top50sales_withcount](https://us1.discourse-cdn.com/elastic/original/3X/c/0/c0346a133ec3ab504460996c276c2fcc154bba95.PNG) This first screen shot shows a visualization of the top 50 most common values found in the vendor\_id term. You can see that according to the visualization, the vendor\_id "1060" appears most often, with a count of 135. ![vendorsales_query](https://us1.discourse-cdn.com/elastic/original/3X/1/7/176a7d82a86cd32b98994e8f59d29ab79b496383.PNG) In this picture, you can see I'm querying the index directly to try to see which logs have a vendor\_id term equal to the value "1060". Again, this search yields 135 hits. Now to the part I'm confused about: ![top10sales_withcount](https://us1.discourse-cdn.com/elastic/original/3X/4/7/47712972104aeea92afe0d1c0654de546c36b72d.PNG) Here you can see a visualization on the same index with the all the same parameters, barring one; the size of the terms to order by. When hovering over the bucketed term corresponding to "1060" there, it reports a count of only "115"! I'm confused where the missing 20 counts went. This behavior isn't only limited to this specific vendor\_id term, "1060" or just for these two size values. The list is ordered differently yet again for 25 terms. See below: ![top25sales](https://us1.discourse-cdn.com/elastic/original/3X/b/2/b2fe4b0b5b58d61538a39611464338fb112407b1.PNG) I must admit this seems rather strange to me. Have I overlooked something about how visualizations work? Thanks again for your time and consideration Lee. Regards, Frank ---
**Author:** ![LeeDr](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leedr/32/9289_2.png) [@LeeDr](https://discuss.elastic.co/u/LeeDr)\ **Post date:** [January 29, 2018, 3:53pm UTC](https://discuss.elastic.co/t/confusion-regarding-unstable-bucket-ing-in-kibana-visualizations/117237/4 "2018-01-29T15:53:55Z")
I tried a similar chart but didn't reproduce your problem. But I know I have seen other posts like this. Here's one; > [@Different Results Based on Aggregation Size](https://discuss.elastic.co/t/different-results-based-on-aggregation-size/56202/4): > > @Shane_Daniel, I agree that the results that you mentioned do sound suspect, and would like to investigate it further, but need to be able to reproduce the issue. I tried to recreate what you described above. I assume it's because the demo data changed since you set this up. (see image to make sure I didn't screw anything up) In it, they used the advanced JSON input field to set the `"shard_size": 0` but I'm trying that and it's giving me an error. I'm still looking into it. ---
**Author:** ![LeeDr](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leedr/32/9289_2.png) [@LeeDr](https://discuss.elastic.co/u/LeeDr)\ **Post date:** [January 29, 2018, 4:00pm UTC](https://discuss.elastic.co/t/confusion-regarding-unstable-bucket-ing-in-kibana-visualizations/117237/5 "2018-01-29T16:00:06Z")
Here's another similar post; > [@Average value change when filtering or changing size](https://discuss.elastic.co/t/average-value-change-when-filtering-or-changing-size/107587/2): > > This sounds like the effect of merging the results from independent shards into a single response, which is generally considered an approximation and with smaller result sizes (when compared to the cardinality) can sometimes result in inaccurate results. Check out [https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-bucket-terms-aggregation.html#\_shard\_size\_3](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-bucket-terms-aggregation.html#_shard_size_3) I suggest using the shard\_size parameter in your terms aggregation to ensure that it pulls back enough ter… I got an error when I tried shard\_size:0 (thinking 0 makes it max out somehow?), but seems to work if I set it large. Give that a try and see if you get the correct counts with it. Looks like this; ![image](https://us1.discourse-cdn.com/elastic/original/3X/1/e/1eb377777f55564589e6da4c36b7cfeff7b8ca75.png) ---
**Author:** ![frankottey](https://avatars.discourse-cdn.com/v4/letter/f/5fc32e/32.png) [@frankottey](https://discuss.elastic.co/u/frankottey)\ **Post date:** [January 29, 2018, 4:49pm UTC](https://discuss.elastic.co/t/confusion-regarding-unstable-bucket-ing-in-kibana-visualizations/117237/6 "2018-01-29T16:49:27Z")
> [@LeeDr](#): > > Give that a try and see if you get the correct counts with it. Hi Lee, Wow. Interesting. Thanks for that tip and I apologize - it appears had I read your initial documentation a little bit closer, I might have expected this. Is there any chance that the 6.1 or later versions of the documentation have a page for "Approximate Aggregations", like the 2.x version of the documentation does? If so, is it possible to tag or link aggregations that are approximate with such a page, or group them together? Or are all bucket aggregations by their nature approximate and I clearly just missed this? For those who come after me, to make it clear: **It appears that term aggregation is an operation that Elasticsearch approximates in order to improve latency**. It appears there is indeed a positive impact from increasing the shard size with regards to search accuracy (assuming you can trade off the compute cost). One of the things I'm still a little confused about is by how much the shard size affects results. By default, according to the documentation, Elasticsearch indices are only created with five shards. Yet I observed that when viewing the top 50 aggregations of the vendor\_id term, the results did not stabilize until the shard\_size value in the advanced JSON field was around 400 or so. I would have assumed, given that detail, that the values would be completely stable at { "shard\_size" = 5 } and beyond... Thank you for your time and consideration Lee, Regards, Frank ---
**Author:** ![LeeDr](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leedr/32/9289_2.png) [@LeeDr](https://discuss.elastic.co/u/LeeDr)\ **Post date:** [January 29, 2018, 7:54pm UTC](https://discuss.elastic.co/t/confusion-regarding-unstable-bucket-ing-in-kibana-visualizations/117237/7 "2018-01-29T19:54:45Z")
Here's the current doc link; [https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-bucket-terms-aggregation.html#search-aggregations-bucket-terms-aggregation-approximate-counts](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-bucket-terms-aggregation.html#search-aggregations-bucket-terms-aggregation-approximate-counts) I think the `shard_size` name is a little misleading. I'm thinking of it as `terms_per_shard`; "When defined, it will determine how many terms the coordinating node will request from each shard." ---
**Author:** ![frankottey](https://avatars.discourse-cdn.com/v4/letter/f/5fc32e/32.png) [@frankottey](https://discuss.elastic.co/u/frankottey)\ **Post date:** [January 29, 2018, 8:52pm UTC](https://discuss.elastic.co/t/confusion-regarding-unstable-bucket-ing-in-kibana-visualizations/117237/8 "2018-01-29T20:52:16Z")
Lee, Thank you for being so patient and pointing me right to the documentation line. That makes absolutely perfect sense. I'm accepting your suggestion to increase the shard\_size as the answer to my issue. I'm also "liking" your comment regarding the meaning of shard\_size. You have been very kind and helpful to a newcomer like myself. Regards, Frank ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [February 26, 2018, 8:52pm UTC](https://discuss.elastic.co/t/confusion-regarding-unstable-bucket-ing-in-kibana-visualizations/117237/9 "2018-02-26T20:52:28Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.