Paging through data

Elastic Stack Elasticsearch
1

I have what I thought was a simple requirement, but am now stumped.

In some cases users call our data and ingest into their own systems, normally we do something like this

The url that calls our application may say something like page=1, page=2

When it hist our application we formulate a query that uses start and total

start 1, total 100
start 101, total 100

That works until I hit the 10,000 limit, so I can't do
start 10001, total 100

This is not for realtime search. Each page would only contain 750 or so records, it won't happen very often (once a day for say 100 pages\calls)

I understand the issue, but not what would be the best solution

  1. change the upper limit, although I know it is not advise
  2. implement scrolling, would require changes to the build of my application for these edge cases

Does anyone have any thoughts on this (I am expecting to have to do 2)

Thanks

Grant

2

Scrolling is generally how you should handle this use case. Changing the upper limit is possible, but I'll explain why it's not advised as well... If you search for "start 10000, total 100", Elasticsearch actually has to go find the first 10100 results and then basically skips the first 10000 in the return. You can imagine as you start increasing that size up to even higher numbers, those higher Elasticsearch will have even more work to do. The limit is put in place to just make developers think "well, this is probably the type of situation where I should use scroll."

Are you running into troubles actually using scroll here?

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.