Random access pagination with search_after on Elasticsearch

Elastic Stack Elasticsearch
1

There are more than 10 thousand documents in my index, but I cannot access all documents with search. I may also have performance problems with the scroll API. I found a method on how to overcome this with search_after in this article.

With this article, I can list all documents without using function score, but it seems that pagination cannot be done using a query. Is there a correct way to search and page through all content?

I want to see the documents by filtering them according to some conditions, but while there should be more than 10k results as a result of filtering and search API, there is max 10k documents. While navigating between pages, it is necessary to be able to move to the 7th page while on the 3rd page. At this point, search_after or scroll are not useful.

2

Welcome

Have a look at The search API | Elasticsearch Guide [8.11] | Elastic

About deep pagination, you can use:

  • the size and from parameters to display by default up to 10000 records to your users. If you want to change this limit, you can change index.max_result_window setting but be aware of the consequences (ie memory).
  • the search after feature to do deep pagination.
  • the Scroll API if you want to extract a resultset to be consumed by another tool later. (Not recommended anymore)

Note that from a end user perspective, going randomly to page 1076 (for example) does not make really sense IMO.
What is your use case?

3

Thanks for your response.
There are more than 100 thousand documents that must come page by page to have a good user experience. That's why I don't want to use Infinite Scroll. It isn't very likely from the end user perspective, but I still think it would be healthier in terms of UX. I cannot transition the pages as I want with the search_after option in your second suggestion, I have tried it before. I will try your first suggestion, but I think we cannot understand what its effects will be in terms of performance.
Thanks again for your response.

4

But what will make the user think that going to page XYZ will help him to find what he wants?

When you are using Google for example, how many times are you going to page XYZ? I bet almost never.
Because you are expecting to have the most relevant information on the first page.

That's the same idea with Elasticsearch.

What kind of data are you indexing? What is the use case?

5

Our customers expect us to list all the content on their platforms because they will do actions such as reviewing, updating, and deleting the content one by one.

They want all content to be accessible and I want to offer them a free environment.. As you said, it is a very low probability, but I am trying to move towards expectations.

When searching on Google, I will likely find what I am looking for on the first page, and in my project, they will want to browse through the pages freely and make reviews.

6

I see...

I think that this usage has a human limit. May be you can just increase the index.max_result_window setting to 100000?

7

Thank you, I will try.

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.