We have the following situation on our hands:
We have 28.000.000 + documents in an index.
There is a background process running that is adding new documents to the index and it is maybe also changing existing documents.
We have a page where the search return 10 results after let's say 20 minutes the hit a "LOAD MORE" button that loads 10 more results below the existing ones based on the same query but with different from-to parameters.
It does seem to occur fairly often that the new documents get ahead of the previously relevant documents and the documents returned in the second batch produce duplicates in the results.
We wish to create a consistent interface.
I've read the guide and the obvious choice would be to use ES scrolls however it is clearly stated that scrolls should never be used to serve user requests for I guess obvious reasons.
Any suggestions hot to keep the result list consistent with the time the actual search was executed?
- User executed a search, 10 result are loaded
- User hits load more button after 2 hours
- The second batch of results should be loaded from the previous query regardless of the changes a made to the background process in the meantime