For exporting data shoud we use scroll or pit with search after?

[Divit_Sharma]

For exporting data shoud we use scroll or pit with search after? Documentation says not to use scroll for deep pagination.

Assume that the data which we are querying would not change.

[Christian_Dahlqvist]

I guess it will depend on which version you are using, but for Elasticsearch 8.4 the recommendation is:

We no longer recommend using the scroll API for deep pagination. If you need to preserve the index state while paging through more than 10,000 hits, use the search_after parameter with a point in time (PIT).

[Divit_Sharma]

@dadoonet Sorry for tagging you but I just wanted to confirm this. I found your reply to below saying that _scroll is preferred over search_after for pulling whole dataset. Does it still apply ?
The documentation has not mentioned this clearly.

Recommendation to use search after instead of scrolling

Hello everybody If you want to read an index completely, you can read in the documentation that the search after function should be used from 10,000 documents (instead of scroll). We have now tested this and unfortunately we have noticed a slowdown by a factor of 10. we sort with the _id meta field. If we use a technical id as a number for sorting, the speed is the same as when scrolling. unfortunately we do not have a technical number field in our use case. why is sorting with a text field …

It still makes sense to me to use _scroll since for search_after I still need to move through the hits to find the sort value in the last document whereas in _scroll I just use the scroll_id for the next batch.

[dadoonet (David Pilato)]

What @Christian_Dahlqvist said is 100% correct.

I should edit my old answer I guess.

[Divit_Sharma]

Thank you @dadoonet

I also had another query about _scroll

Sliced scroll returning more hits than normal search (without slice)

In short, normal search hits < scroll slice 1 hits + scroll slice 2 hits When I add up sliced scroll hits, it is more than total no of documents returned from single search. Normal search query GET index*/_search { "track_total_hits": true, "sort": [ { "@timestamp": { "order": "asc", "unmapped_type": "boolean" } } ], "_source": false, "query": { "bool": { "must": [], "filter": [ { "range": { "@timesta…