Paginating result set greater than 10000 (with aggregations) - Possible options

Elastic Stack Elasticsearch
5

I am trying to cover up the use case wherein a user does take the pain of navigating to 501th page!

I see. So no search here. You are basically searching with a match_all query and you only have a filter by category, right?

I believe you have few choices here:

  • Blocking the user after page 500 explaining that he should refine its search.
  • Increasing the default value index.max_result_window but as you know already this comes with a cost. Also the deeper you go, the slower it will be
  • Unless you use search_after feature where elasticsearch can do some optimizations but you can't really go back IMO.
  • scroll might be something to consider but it has some drawbacks: keeping segments around until all scrolls has been released, if the user take too much time on a page before scrolling again he will loose the scroll id and will restart, aggs: you need to keep them somewhere in your app layer, going back is not allowed either.

Redesigning the result page might be a better choice IMO but I don't know if it's doable in your context. I like the faceted navigation pattern which is used everywhere to help the user finding the best data for him without having to go 100s of pages.

My 0.05 cents

2 Likes