Hello people,
My query is retrieving 10000 records from elasticSearch in 1 hit.
I used pagination (from & size) but its giving me only specified records.
I want 10000 records but in a batch size of 50.
(I am not able to understand how scroll is used with from & size.)
How can we accomplish that one ?
When you retrieve documents through scroll, each new invocation retrieves the next 'page' of data. You cannot specify a from or to parameter. Scroll allows to retrieve any number of documents and is typically used to export data from an index. It should not be used for casual retrieval of a few pages and not be the standard pagination mechanism for the display of interactive pages. Each time a scroll is used, the affected shards are not allowed to perform background merges until the scroll has been released.
Pagination with from and to on the other hand should only be used to retrieve the first few pages of a data set (e.g. up to maybe 10 pages of size 50), as pagination becomes very expensive with increasing depth.
You may also want to investigate if search after suits the needs of the application. This mechanism is a lot less taxing on the index than the scroll mechanism and can be used to retrieve larger amounts of data in an interactive setup.
1 Like
Yes it will give the records as per the size in 1st hit, but what about other records ?
And how do we get number of records retrieved from the response ?
Below is my search response :
SearchResponse response = highLevelRestClient.search(searchRequest, RequestOptions.DEFAULT);
When you use the scroll mechanism, the first request returns a scroll_id in the response. You pass this value on the next request back to Elasticsearch for the next slice of data. This will return another scroll_id.
1 Like
Got it. Thanx
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.