Paginate elastic results on specific field

Elastic Stack Elasticsearch
1

Hello !
I have data like this in my elasticsearch :

{
    doc_name : 'Name 1',
    member_id: 'id1'
}
{
    doc_name : 'Name 2',
    member_id: 'id1'
},
{
    doc_name : 'Name 3',
    member_id: 'id2'
}

I have my own scoring function which is very important so I need to retrieve data in the correct order. No problem here but I now want to limit my results by number of members, to get only the ordered results but for only the first 10 members. Then make a from clause to make a load more button.

I'm not even sure it is possible... If you have any idea on how I can make something like this, please feel free !

Thank you !

2

I'm not quite sure I understand your requirements. Are you saying you want the hits sorted using your custom scoring function, but that you only want to see one hit per unique member_id? In that case, field collapsing is exactly what you are looking for: https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-collapse.html

3

Thanks for your answer ! That's not exactly that.

I want to retrieve my hits ordered by my score, and stop as I reach for example 10 distinct member_ids. My limit would not be on the document count but on a specific field.
More difficult : I want to add a Load More button, so that I could retrieve the next hits and so on.

I'm not quite sure I can perform something like this with the field collapsing...

Thank you

4

What is it that you want to retrieve for each hit? Just the member_ids? Or all documents that belong to each of the top 10 member_ids?

What would happen with a document of a member in the top 10 that scores lower than a document of member #11? Would it be returned when you click " Load More"?

Sorry for these questions, but I'm trying to understand what it is that you're trying to do :slight_smile:

5

No problem thank you for your time !

It's hard to explain haha ! I'll try to make an example. Supposing that without limit, elastic gets me as an answer the documents below :

{
    doc_name : 'Name 1',
    member_id: 'id1'
}
{
    doc_name : 'Name 4',
    member_id: 'id4'
},
{
    doc_name : 'Name 2',
    member_id: 'id1'
},
{
    doc_name : 'Name 5',
    member_id: 'id3'
}
{
    doc_name : 'Name 3',
    member_id: 'id2'
}

I'm also assuming it's returned by a bool query in a function score array, so ordered as I wish.

In my case, I want to get my results for the first 3 member_ids for example. My elastic should give me :

{
    doc_name : 'Name 1',
    member_id: 'id1'
}
{
    doc_name : 'Name 4',
    member_id: 'id4'
},
{
    doc_name : 'Name 2',
    member_id: 'id1'
},
{
    doc_name : 'Name 5',
    member_id: 'id3'
}

Then, I want to have sort of a "pagination" to be able to add a Load More button, which sould result in the next hits for the 3 next member_ids (in this case only the last result, but in reality, a lot more.

Hope I'm understandable ! English is not my native language :wink:

6

English is not my native language either, but I think we're doing just fine :wink:

I don't think Elasticsearch offers you something out of the box without you having to do some bookkeeping on the client side. I do think field collapsing on the member_id field can be helpful, but it's not the complete solution for your case.

You will have to use the from and size parameters to paginate and get to pages of exactly 10 unique member_ids per result page. Tracking the number of members returned and remembering where you are in the pagination set is something you need to do yourself in the client application.

7

And using from and size to get 10 unique member_id per result page is something I can do with field collapsing ?

Thank you,

8

No, you will have to keep paginating until you get to 10 unique member_ids. Field collapsing will just make it a bit easier to keep track of how many IDs are being returned.

9

So my only way to achieve that is to make it client-side, do I understand correctly ?

10

Yes, I think that's the way to go, unless someone else has a better idea.

11

Ok ! That's what I thought. Thanks for your help, I'm totally new to elasticsearch and it can make so many things I just had to know !

Than you very much and have a nice day,
Dagan

12

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.