Pagination with HighLevelRestClient

Elastic Stack Elasticsearch
1

Hello,
I would like to use HighLevelRestClient to fetch all results matching a given query using pagination. Now, checking on the rest api I can found that scroll api have been replaced by search_after.
So I checked the HighLevelRestClient documentation in order to understand how to implement pagination but there I found documentation only for the scroll api. So which is the right implementation? I am not performing this query in order to show results to the user but to process a batch of data. Of course while paginating I can still receive data to write or to remove from the index so I would like to keep a coherent view while paginating. Can you please tell me which is the bast way to paginate? Also, I made an experiment with search_after (using the HighLevelRestClient) and it´s working but I don´t see any PIT (point in time) parameter.
Thanks a lot!

2

This sounds unrelated to our Enterprise Search or App Search products. Reassigning to elasticsearch.

3

Thanks @JasonStoltz, sorry my bad, I chose the wrong category :frowning:

4

Yeah. True. See also the discussion at How to make Point In Time API request using JAVA REST High Level Client

5

You can use:

  • the size and from parameters to display by default up to 10000 records to your users. If you want to change this limit, you can change index.max_result_window setting but be aware of the consequences (ie memory).
  • the search after feature to do deep pagination.
  • the Scroll API if you want to extract a resultset to be consumed by another tool later.

The same applies to the rest client :wink:

6

Hello @dadoonet, thanks for your reply. Not sure I understood the part related to PIT, the link points to this thread again. Did you mean another thread? In that case, could you please fix it?

7

Also, @dadoonet which is your advice? Which is the best practice?
I need to go over all the results for a query (it's a percolate query, just to provide you one more detail) and I would like the pagination to be consistent even if in the meantime changes happen. Should I use scroll or search after from the HighLevelRestClient?
Thanks!

8

:scream: Fixed

9

I'd use the scroll API.