# Terms aggregation bucket Sorting is not working when using with partition **URL:** \ **Category:** Elasticsearch\ **Created:** [March 1, 2022, 12:08pm UTC](https://discuss.elastic.co/t/terms-aggregation-bucket-sorting-is-not-working-when-using-with-partition/298502 "2022-03-01T12:08:35Z")\ **Posts on this page:** 14\ **Page:** 1
**Author:** ![truptir](https://avatars.discourse-cdn.com/v4/letter/t/58f4c7/32.png) [@truptir](https://discuss.elastic.co/u/truptir)\ **Post date:** [March 1, 2022, 12:08pm UTC](https://discuss.elastic.co/t/terms-aggregation-bucket-sorting-is-not-working-when-using-with-partition/298502/1 "2022-03-01T12:08:35Z")
I am trying to do TermAggregation + Sort order(by \_count) + pagination(using partition). But resulting bucket partitions are not sorted properly. Without using partitions, sorting is working properly. Here is my elastic Query, ```auto GET inventory/_search?filter_path=aggregations { "aggs": { "totalPacks": { "terms": { "field": "sellerId.keyword", "min_doc_count": 1, "shard_min_doc_count": 0, "show_term_doc_count_error": false, "include": { "partition": 0, "num_partitions": 10 }, "size": 10, "order": { "_count": "asc" } } } } } ``` this gives output , partition: 0 ```auto "buckets" : [ { "key" : "305860", "doc_count" : 3 }, { "key" : "13101", "doc_count" : 10 } ``` partition: 2 ```auto "buckets" : [ { "key" : "05702", "doc_count" : 4 }, { "key" : "17188", "doc_count" : 42 } ] ``` doc\_count: 4 is coming in 1st but should come in 0th partition doc\_count: 10 is coming on 0th but should come in 1ST partition. ---
**Author:** ![Mark\_Harwood](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mark_harwood/32/10538_2.png) [@Mark\_Harwood](https://discuss.elastic.co/u/Mark_Harwood)\ **Post date:** [March 1, 2022, 12:59pm UTC](https://discuss.elastic.co/t/terms-aggregation-bucket-sorting-is-not-working-when-using-with-partition/298502/2 "2022-03-01T12:59:12Z")
This is working as designed. Sort order is valid within partitions, not across them. It’s a coping strategy for when your data is distributed across lots of machines which makes it difficult to perform certain operations in a single search. It relies on your client application stitching together results from multiple partitions where appropriate. ---
**Author:** ![truptir](https://avatars.discourse-cdn.com/v4/letter/t/58f4c7/32.png) [@truptir](https://discuss.elastic.co/u/truptir)\ **Post date:** [March 1, 2022, 1:07pm UTC](https://discuss.elastic.co/t/terms-aggregation-bucket-sorting-is-not-working-when-using-with-partition/298502/3 "2022-03-01T13:07:54Z")
it is partitioning in which order? And if we need to apply sort first and then partition the data. Is there any approach? ---
**Author:** ![Mark\_Harwood](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mark_harwood/32/10538_2.png) [@Mark\_Harwood](https://discuss.elastic.co/u/Mark_Harwood)\ **Post date:** [March 1, 2022, 2:36pm UTC](https://discuss.elastic.co/t/terms-aggregation-bucket-sorting-is-not-working-when-using-with-partition/298502/4 "2022-03-01T14:36:15Z")
[This comment](https://discuss.elastic.co/t/elasticsearch-terms-aggregation-with-partition-does-not-honor-the-size-value/271294/4) with links may be of use ---
**Author:** ![truptir](https://avatars.discourse-cdn.com/v4/letter/t/58f4c7/32.png) [@truptir](https://discuss.elastic.co/u/truptir)\ **Post date:** [March 2, 2022, 1:07pm UTC](https://discuss.elastic.co/t/terms-aggregation-bucket-sorting-is-not-working-when-using-with-partition/298502/5 "2022-03-02T13:07:10Z")
Hi @Mark_Harwood, I understood the issue. But Is there a way to sort data across all partitions? Or can u suggest any other way using that we can achieve Aggregation + sorting across all data+ pagination? We are wondering how other big apps that use ES, manage the sort order with pagination. ---
**Author:** ![Mark\_Harwood](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mark_harwood/32/10538_2.png) [@Mark\_Harwood](https://discuss.elastic.co/u/Mark_Harwood)\ **Post date:** [March 2, 2022, 1:59pm UTC](https://discuss.elastic.co/t/terms-aggregation-bucket-sorting-is-not-working-when-using-with-partition/298502/6 "2022-03-02T13:59:20Z")
The challenge is data locality. Unless you have all data related to a seller kept on the same shard it’s harder to reason about each seller. Time-based indices make this harder. The fact you’re sorting by lowest doc count makes this even harder. If you can put all docs related to the same seller in one shard there’s hope. “Custom routing” or the “transform” api are 2 ways to ensure information related to each seller is available in only one shard. ---
**Author:** ![Tomo\_M](https://avatars.discourse-cdn.com/v4/letter/t/848f3c/32.png) [@Tomo\_M](https://discuss.elastic.co/u/Tomo_M)\ **Post date:** [March 3, 2022, 12:56pm UTC](https://discuss.elastic.co/t/terms-aggregation-bucket-sorting-is-not-working-when-using-with-partition/298502/7 "2022-03-03T12:56:25Z")
If you need ` "_count": "asc"`, what will happen if using [rare terms aggregation](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-bucket-rare-terms-aggregation.html)? Do you still need pagination even with rare terms aggregation?? ---
**Author:** ![truptir](https://avatars.discourse-cdn.com/v4/letter/t/58f4c7/32.png) [@truptir](https://discuss.elastic.co/u/truptir)\ **Post date:** [March 22, 2022, 12:36pm UTC](https://discuss.elastic.co/t/terms-aggregation-bucket-sorting-is-not-working-when-using-with-partition/298502/8 "2022-03-22T12:36:04Z")
> [@Mark\_Harwood](#): > > Unless you have all data related to a seller kept on the same shard it’s harder to reason about each seller. Hi @Mark_Harwood, I have 1 shard and 1 replica for my index then it should work right? Descending order on count is also not working even I have only one shard for my index. Index Setting: ```auto "number_of_shards": "1", "number_of_replicas": "1" ``` ---
**Author:** ![Mark\_Harwood](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mark_harwood/32/10538_2.png) [@Mark\_Harwood](https://discuss.elastic.co/u/Mark_Harwood)\ **Post date:** [March 22, 2022, 5:49pm UTC](https://discuss.elastic.co/t/terms-aggregation-bucket-sorting-is-not-working-when-using-with-partition/298502/9 "2022-03-22T17:49:53Z")
> [@truptir](#): > > Descending order on count is also not working In what way is it “not working”? Can you share some outputs ---
**Author:** ![truptir](https://avatars.discourse-cdn.com/v4/letter/t/58f4c7/32.png) [@truptir](https://discuss.elastic.co/u/truptir)\ **Post date:** [March 23, 2022, 5:31am UTC](https://discuss.elastic.co/t/terms-aggregation-bucket-sorting-is-not-working-when-using-with-partition/298502/10 "2022-03-23T05:31:49Z")
> [@Mark\_Harwood](#): > > In what way is it “not working”? I meant Descending sorting on count is not working across all data. I m looking for a way that returns descending sorted data on \_count across all partitions. As per your comment, I understood Sorting may not work across all data if data resides in different shards/machines. But here my index configuration is 1 shard. please suggest if this is possible by changing any configuration. Output: For DESC order on \_count Partition: 0 ```auto "buckets" : [ { "key" : "34950", "doc_count" : 11 }, { "key" : "28747", "doc_count" : 10 } ] ``` Partition:1 ```auto "buckets" : [ { "key" : "29194", "doc_count" : 65 }, { "key" : "16518", "doc_count" : 36 } ] ``` ---
**Author:** ![Mark\_Harwood](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mark_harwood/32/10538_2.png) [@Mark\_Harwood](https://discuss.elastic.co/u/Mark_Harwood)\ **Post date:** [March 23, 2022, 5:00pm UTC](https://discuss.elastic.co/t/terms-aggregation-bucket-sorting-is-not-working-when-using-with-partition/298502/11 "2022-03-23T17:00:13Z")
> [@truptir](#): > > I m looking for a way that returns descending sorted data on \_count across all partitions. That could be done by just not using partitions, no? ---
**Author:** ![truptir](https://avatars.discourse-cdn.com/v4/letter/t/58f4c7/32.png) [@truptir](https://discuss.elastic.co/u/truptir)\ **Post date:** [March 25, 2022, 6:35am UTC](https://discuss.elastic.co/t/terms-aggregation-bucket-sorting-is-not-working-when-using-with-partition/298502/12 "2022-03-25T06:35:58Z")
But I want a Paginated bucket. result query may return more than 1 lakhs bucket but at a time it should return 15 buckets this is the reason I am using partitions. ---
**Author:** ![Mark\_Harwood](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mark_harwood/32/10538_2.png) [@Mark\_Harwood](https://discuss.elastic.co/u/Mark_Harwood)\ **Post date:** [March 25, 2022, 7:47am UTC](https://discuss.elastic.co/t/terms-aggregation-bucket-sorting-is-not-working-when-using-with-partition/298502/13 "2022-03-25T07:47:04Z")
Deep pagination on a derived value is tough because the deeper you go into the smaller values you have to hold all the other keys and derived values in memory just to figure out where you are in the global order. Maybe the solution is turn the derived value into a concrete one using the “transform” api to create a derived index holding the totals for each key. This can be queried and sorted on the total field using [search\_after](https://www.elastic.co/guide/en/elasticsearch/reference/current/paginate-search-results.html#search-after) param. This can be implemented efficiently because it requires less memory but does not support direct indexing into the sort position (eg randomly picking a page number from the results ) ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [April 22, 2022, 7:47am UTC](https://discuss.elastic.co/t/terms-aggregation-bucket-sorting-is-not-working-when-using-with-partition/298502/14 "2022-04-22T07:47:18Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.