I am trying to cover up the use case wherein a user does take the pain of navigating to 501th page!
I see. So no search here. You are basically searching with a match_all query and you only have a filter by category, right?
I believe you have few choices here:
- Blocking the user after page 500 explaining that he should refine its search.
- Increasing the default value
index.max_result_windowbut as you know already this comes with a cost. Also the deeper you go, the slower it will be - Unless you use search_after feature where elasticsearch can do some optimizations but you can't really go back IMO.
- scroll might be something to consider but it has some drawbacks: keeping segments around until all scrolls has been released, if the user take too much time on a page before scrolling again he will loose the scroll id and will restart, aggs: you need to keep them somewhere in your app layer, going back is not allowed either.
Redesigning the result page might be a better choice IMO but I don't know if it's doable in your context. I like the faceted navigation pattern which is used everywhere to help the user finding the best data for him without having to go 100s of pages.
My 0.05 cents