Paged Search Larger than 10,000 Records

Elastic Stack Elasticsearch
1

We are currently using from and size pagination in our Elasticsearch results of ~1,000 records on average. We recently ran into some instances of ~15,000 records and expect some more of similar magnitude, resulting in the 10,000 record cap causing issues.

Users can navigate to specific pages and change the size of the pages in real time, which search after and scroll search don't seem to support. Would increasing the value for index.max_result_window to 20,000 or 30,000 and continue to use from and size pagination be our best option in this case, or is there a more efficient method?

Paginate search results | Elasticsearch Guide [8.17] | Elastic

2

Hi @Devan,

I believe that there is no better way than search_after. I know some applications had to drop the feature of adjustable page size because of it.

Alternatively, you can think of adjusting relevance metrics for your search so that the user does not have to page so far to find what they are looking for.

1 Like
3

search_after appears to require parameters from the previous page's results, so it wouldn't allow for users to jump between pages right?

The groups of items ranging anywhere from ~100 to ~15,000 are already filtered down. If being able to navigate to any page by entering a page number is a requirement, no more than 100 items at a time, is that only really doable with the "from" and "size" parameters?

4

If you absolutely need that level of random access, you probably have to use from/size and increase your context window.

The only other suggestion I have is to look at the requirements of your search application. Is randomly paginating that deep truly required or can your application for example send in additional filters to return smaller overall result sets? It really depends on your use case and what you're trying to achieve. For many search use cases, there is no need to go that deep into search results.

1 Like
5

Hi! :waving_hand:

When your result sets go beyond 10,000 documents, using from + size starts to introduce performance and memory overhead — especially with high from values like 15000.

:white_check_mark: Recommended Options

  1. Use search_after
    Great for sequential pagination. Fast, lightweight, and doesn’t scan skipped results like from does.
  2. Use search_after + Point-in-Time (PIT)
    Ensures consistent results even if the index is being updated. Start with:

json

POST /my-index/_search/point_in_time?keep_alive=1m
  1. Hybrid Strategy
    Use from + size for the first ~2,000 results, then switch to search_after for deeper pages.

:warning: About index.max_result_window

You can increase it like this:

json

PUT my-index/_settings
{
  "index": {
    "max_result_window": 30000
  }
}

But this is not scalable and can put pressure on memory. It’s OK for occasional deep paging, but not recommended for high traffic patterns.

Let me know if you'd like help building a search_after example or merging pagination logic on the frontend.

—
Shared by an Elastic Stack user based on real-world cluster use cases.

1 Like