# Elastic Search \> @SearchUI \> group with sort

**URL:** https://discuss.elastic.co/t/elastic-search-searchui-group-with-sort/327154
**Category:** Elastic Search
**Tags:** elastic-app-search
**Created:** [March 7, 2023, 8:12am UTC](https://discuss.elastic.co/t/elastic-search-searchui-group-with-sort/327154 "2023-03-07T08:12:05Z")
**Posts on this page:** 17
**Page:** 1

<div class="post-metadata">

### Author: ![parliament718](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/parliament718/32/117243_2.png) [@parliament718](https://discuss.elastic.co/u/parliament718)
#### Post date: [March 7, 2023, 8:12am UTC](https://discuss.elastic.co/t/elastic-search-searchui-group-with-sort/327154/1 "2023-03-07T08:12:05Z")

</div>

I've been trying for days to get group+sort working with @search-ui, with either the app-search connector or the elasticsearch connector with limited success with both.

Originally I used the app-search connector and set the `group` on the query:

```auto
group: {
      collapse: true,
      field: 'card_code_rarity',
},

```

This almost gives me the result I want, with each search result containing a `_group` property with the documents in the group. The only additional thing I need is to sort by the count of these documents. If the following worked it would be fantastic:

```auto
group: {
      collapse: true,
      field: 'card_code_rarity',
},
sort: [
    { '_group.length': 'desc' },
],

```

But it doesnt. After hitting non-step dead-ends I decided to try to Elasticsearch connector with search-ui.

I can get some form of a usable end result with the following in POSTMAN:

```auto
aggs: {
            card_code_rarity: {
               terms: {
                  field: "card_code_rarity",
                  order: {
                     "_count": "desc"
                  }
               },
               aggs: {
                  my_docs: {
                     top_hits: {
                        sort: [
                           {
                              "selling_price": { "order": "desc" }
                           }
                        ]
                     }
                  }
               }
            }
         },

```

However this part of the query gets completely omitted by the Elasticsearch connector, it seems to not have support for this.... and it wont even forward the params as-is.

This connector also doesn't seem to have a `beforeSearch` hook like the app-search one, and it's not easy to extend it since it doesnt export `handleSearchRequest()`. So what to do?

Ideally, I can go back to using AppSearch connector and get the sorting to work somehow.

Otherwise how can I get the aggregate query to work with the Elasticsearch connector?

Actually, would be nice to know how to do it with either connector in case I need to switch again for some other limitation.

---

<div class="post-metadata">

### Author: ![parliament718](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/parliament718/32/117243_2.png) [@parliament718](https://discuss.elastic.co/u/parliament718)
#### Post date: [March 7, 2023, 11:32am UTC](https://discuss.elastic.co/t/elastic-search-searchui-group-with-sort/327154/2 "2023-03-07T11:32:59Z")

</div>

Looks like I can achieve the _query_ I need using the second param handler to the connector like so:

```auto
const connector = new ElasticsearchAPIConnector(
  { ... }, 
  (requestBody, requestState, queryConfig) => {   
    requestBody.aggs = queryConfig.aggs;

    return requestBody;
  }
);

```

This probably shouldnt be neccesary but I do get the expected results in the Chrome network tab.

However, yet another blocker here is I'm unable to extract the **full** Elasticsearch response from the connector, and there is nothing I can use in the `SearchDriver`'s `state` object.

---

<div class="post-metadata">

### Author: ![joemcelroy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/joemcelroy/32/103757_2.png) [@joemcelroy](https://discuss.elastic.co/u/joemcelroy)
#### Post date: [March 7, 2023, 12:04pm UTC](https://discuss.elastic.co/t/elastic-search-searchui-group-with-sort/327154/3 "2023-03-07T12:04:39Z")

</div>

hey @parliament718

You should be able to get the raw elasticsearch response for each result via the `_meta.rawHit`

> <https://github.com/elastic/search-ui/blob/07ee4c4c2b1ac707a38813464f4df8933380cddb/packages/search-ui-elasticsearch-connector/src/handlers/common.ts#L26>

Joe

---

<div class="post-metadata">

### Author: ![parliament718](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/parliament718/32/117243_2.png) [@parliament718](https://discuss.elastic.co/u/parliament718)
#### Post date: [March 7, 2023, 12:17pm UTC](https://discuss.elastic.co/t/elastic-search-searchui-group-with-sort/327154/4 "2023-03-07T12:17:23Z")

</div>

@joemcelroy  
Thanks for the reply.

Unfortunately that is not enough. I need the `aggregations` value at the root of the response. That is a siblings of `hits`.

---

<div class="post-metadata">

### Author: ![parliament718](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/parliament718/32/117243_2.png) [@parliament718](https://discuss.elastic.co/u/parliament718)
#### Post date: [March 7, 2023, 12:19pm UTC](https://discuss.elastic.co/t/elastic-search-searchui-group-with-sort/327154/5 "2023-03-07T12:19:39Z")

</div>

I've been digging into the lib source like crazy and it's just so hard to extend because of all the default exports, none of the useful classes are exported.

As such, I can't extend `SearchkitResponseTransformer` which seems to be what I need. No way I see to provide my own

---

<div class="post-metadata">

### Author: ![parliament718](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/parliament718/32/117243_2.png) [@parliament718](https://discuss.elastic.co/u/parliament718)
#### Post date: [March 7, 2023, 1:53pm UTC](https://discuss.elastic.co/t/elastic-search-searchui-group-with-sort/327154/6 "2023-03-07T13:53:07Z")

</div>

Even with some hacks to compiled code to get that property from the response, my implementation is broken in respect to pagination as it will just return the same buckets each time.

Is there a way to revert back to AppSearchConnector where it was returning the "\_group" on each hit and the pagination was working... that was sooooo close to what I need. All I need is to sort the hits by the count of docs in the \_group, like:

```auto
group: {
      collapse: true,
      field: 'card_code_rarity',
},
sort: [
    { '_group.length': 'desc' }, <!-- is there something Im missing to make this work
],

```

I can't believe this is so hard, seems like a common and basic use case.

---

<div class="post-metadata">

### Author: ![joemcelroy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/joemcelroy/32/103757_2.png) [@joemcelroy](https://discuss.elastic.co/u/joemcelroy)
#### Post date: [March 7, 2023, 2:00pm UTC](https://discuss.elastic.co/t/elastic-search-searchui-group-with-sort/327154/7 "2023-03-07T14:00:08Z")

</div>

Yeh elasticsearch-connector wasn't designed to be extensible like that.

Its an API limitation on App Search to allow sorting of the group.

One option could be to fork the elasticsearch-connector and customise.

Alternative is using searchkit directly ([www.searchkit.co](http://www.searchkit.co)). That allows customising the full elasticsearch query and response with request hooks [Search Relevance – Searchkit](https://www.searchkit.co/docs/guides/customising-query#request-hooks)

---

<div class="post-metadata">

### Author: ![parliament718](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/parliament718/32/117243_2.png) [@parliament718](https://discuss.elastic.co/u/parliament718)
#### Post date: [March 7, 2023, 3:36pm UTC](https://discuss.elastic.co/t/elastic-search-searchui-group-with-sort/327154/8 "2023-03-07T15:36:33Z")

</div>

At this point it's not a matter of which connector but whether ES can even do this. After testing all kinds of aggregates my results are as follows:

**Terms Aggregate:**

- Allows sort by \_count ✅
- Gets actual documents ✅
- No pagination ❌

**Composite Aggregate:**

- Allows pagination via "after" ✅
- Doesnt allow sort over \_count. ❌
- Doesnt return actual documents ❌
- Doesnt work with top\_hits ❌

**Bucket Sort Aggregate:**

- Allows to sort by \_count ✅
- Allows pagination via "from" ✅
- Doesn't return actual documents ⚠
- Works with top\_hits to return actual documents ✅

The Bucket Sort Aggregate **appears** to be giving me the desired results but is there some other limitation I should be aware of? Are these results accurate? Frankly it's shocking the state of capabilities after a decade of development.

---

<div class="post-metadata">

### Author: ![parliament718](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/parliament718/32/117243_2.png) [@parliament718](https://discuss.elastic.co/u/parliament718)
#### Post date: [March 7, 2023, 3:37pm UTC](https://discuss.elastic.co/t/elastic-search-searchui-group-with-sort/327154/9 "2023-03-07T15:37:53Z")

</div>

Here's what **appears** to be working for me, a mix of bucket\_sort and top\_hits:

```auto
aggs: {
            my_bucket_sort: {
               terms: {
                  field: 'card_code_rarity',
                  // We can do sort here also
                  size: 1000000 // Keep this size big integer. This keep all possible result in bucket
               },
               aggs: {
                  my_bucket_sort: {
                     bucket_sort: {
                        sort: [
                           {
                              _count: {
                                 order: 'desc'
                              }
                           }
                        ],
                        from: 0, // use this to paginate
                        size: 12 // page size
                     },
                  },
                  //need this to get the actual documents
                  listings: {
                     top_hits: { }
                  }
               }
            }
         },
         size: 0,

```

Is combining bucket\_sort with top\_hits like this a supported query by design or are my results coincidental/unreliable?

---

<div class="post-metadata">

### Author: ![joemcelroy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/joemcelroy/32/103757_2.png) [@joemcelroy](https://discuss.elastic.co/u/joemcelroy)
#### Post date: [March 7, 2023, 4:00pm UTC](https://discuss.elastic.co/t/elastic-search-searchui-group-with-sort/327154/10 "2023-03-07T16:00:49Z")

</div>

Could I ask what are you trying to do / the experience you hope to achieve? I made the assumption you're grouping on a field and you want to show inner results. Using collapse functionality could work for you [Collapse search results | Elasticsearch Guide [8.6] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/collapse-search-results.html)

---

<div class="post-metadata">

### Author: ![parliament718](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/parliament718/32/117243_2.png) [@parliament718](https://discuss.elastic.co/u/parliament718)
#### Post date: [March 7, 2023, 4:36pm UTC](https://discuss.elastic.co/t/elastic-search-searchui-group-with-sort/327154/11 "2023-03-07T16:36:04Z")

</div>

@joemcelroy  
Collapse sounded promising but I'm not getting expected results. If I add:

```auto
"collapse": {
    "field": "card_code_rarity"        
  },

```

All that gives me is the following extra data for each hit in the response:

```auto
"fields": {
      "card_code_rarity": [
         "TDGS-EN038-(UtR)"
       ]
  }

```

I was expecting some sort of bucket with the documents for that rarity. Ok let me try to explain what Im trying to do:

It's a marketplace for cards. There are `Card` objects and `CardListing` objects.

When a `Card` is "listed" for sale, I take the Card object, clone it, add some properties (selling\_price, etc), and store it in the same index.

`card_code_rarity` is **unique** for a `Card` but not for a `CardListing` (since for 3 listings I'm cloning the card with the same card\_code\_rarity)

I want the user to search **all cards** and for each result to see how many listings there are, with the cards having the most listings at the top.

Example Results Display:

- **Card A:** 4 listings from $20 - $50
- **Card B:** 3 listings from $5 - $15
- **Card C:** 0 listings

To do this I want to

- group by `card_code_rarity` to get the listings per card
- sort by the \_count to get most listed cards at the top
- all this should paginate, facet, filter, sort, etc normally at the top level

Thanks again for the help

---

<div class="post-metadata">

### Author: ![parliament718](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/parliament718/32/117243_2.png) [@parliament718](https://discuss.elastic.co/u/parliament718)
#### Post date: [March 7, 2023, 4:47pm UTC](https://discuss.elastic.co/t/elastic-search-searchui-group-with-sort/327154/12 "2023-03-07T16:47:15Z")

</div>

Although I havent fully vetted the results yet, combining bucket\_sort + top\_hits aggregates seems to give expected results though I already see a limitation that `top_hits.size` can only be max 100, meaning only 100 listings per card will be retrieved and display info inaccurate if there are more. This is fine for now, but isnt scalable.

I only dispay a handful of cards at a time due to pagination, so it would be better to retrieve all the listings for them.

---

<div class="post-metadata">

### Author: ![joemcelroy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/joemcelroy/32/103757_2.png) [@joemcelroy](https://discuss.elastic.co/u/joemcelroy)
#### Post date: [March 7, 2023, 4:59pm UTC](https://discuss.elastic.co/t/elastic-search-searchui-group-with-sort/327154/13 "2023-03-07T16:59:30Z")

</div>

have you tried using inner\_hits option aswell with collapse? You can specify the sorting here. Sorry if you have considered this already

> **[Collapse search results | Elasticsearch Guide \[8.6\] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/collapse-search-results.html#expand-collapse-results)**

---

<div class="post-metadata">

### Author: ![joemcelroy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/joemcelroy/32/103757_2.png) [@joemcelroy](https://discuss.elastic.co/u/joemcelroy)
#### Post date: [March 7, 2023, 5:07pm UTC](https://discuss.elastic.co/t/elastic-search-searchui-group-with-sort/327154/14 "2023-03-07T17:07:33Z")

</div>

you might be able to do this in the nested fields approach by indexing the card and card listings being children which have the price

> **[Nested field type | Elasticsearch Guide \[8.6\] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/nested.html)**

---

<div class="post-metadata">

### Author: ![parliament718](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/parliament718/32/117243_2.png) [@parliament718](https://discuss.elastic.co/u/parliament718)
#### Post date: [March 7, 2023, 10:38pm UTC](https://discuss.elastic.co/t/elastic-search-searchui-group-with-sort/327154/15 "2023-03-07T22:38:04Z")

</div>

@joemcelroy Thank you.

The inner\_hits approach seems to give me expected results as far as hierarchy, but I'm still unable to sort by the count of inner\_hits documents.

```auto
 "collapse": {
            "field": "card_code_rarity",
            "inner_hits": {
               "name": "most_recent",
               "size": 5,
               "sort": [{ "selling_price": "desc" }]
            },
            "max_concurrent_group_searches": 4
         },
         sort: [
              <!--- still need a way to sort at this level based on count of inner_hits
         ],

```

If that is supported I would be golden. If not, nesting the documents would represent a significant refactoring, although I have an idea of how that might work

[In this article](https://medium.com/@ak2124454/sort-parent-documents-based-on-nested-child-count-in-elastic-search-580daf24f78c), a constant `weight: 1` is added to each child document and then summed for the count using "mode": "sum". **Is that the best approach if nesting?** Could I do something similar with inner\_hits (adding a weight property to each listing) and not need nesting ?

Thank you

---

<div class="post-metadata">

### Author: ![joemcelroy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/joemcelroy/32/103757_2.png) [@joemcelroy](https://discuss.elastic.co/u/joemcelroy)
#### Post date: [March 8, 2023, 8:39am UTC](https://discuss.elastic.co/t/elastic-search-searchui-group-with-sort/327154/16 "2023-03-08T08:39:10Z")

</div>

heres a long running thread about collapse and being able to sort from the collapsed groups [Add support for collapse sort to specify how to sort groups relative to each other · Issue #45646 · elastic/elasticsearch · GitHub](https://github.com/elastic/elasticsearch/issues/45646) but hasn't yet made it to a release yet ☹

I think you will run into pagination problems and how to integrate this into search ui aswell!

I suggest looking at nested. Searchkit supports nested fields (which search-ui doesn't)

I dont know the best strategy to sort for your example but you can sort on nested documents in this example.

> **[Sort search results | Elasticsearch Guide \[8.6\] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/sort-search-results.html#_nested_sorting_examples)**

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [April 5, 2023, 8:40am UTC](https://discuss.elastic.co/t/elastic-search-searchui-group-with-sort/327154/17 "2023-04-05T08:40:00Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
