- May be. May be not that longer that you can imagine.
- I think it's wrong. The opposite is true. I mean that loading on the coordinated node 10k docs instead of 1m of docs for example will consume much less memory.
Once you get back 10k docs in your application, just stream them to the other process/destination you want.
What kind of error? You mean elasticsearch does not work anymore? Then you need to fix the root of the problem not trying to work around it.
I think you are talking about something going wrong then in your own process, right?
If so, you have to deal with that on your side.
Yes this not how Scroll is working. You never specify the from.
Basically, you run a search request. It returns 10m of docs and you start to scroll the result set from the beginning x docs per x docs.
When you run the next scroll call, it asks for the next x docs you want.
That said, may be this new feature coming in 5.0 could help for your case?
Also this new feature could help to resume a scroll at some point. No date for this feature though. Introduce a new `_last_modified` metafield · Issue #20859 · elastic/elasticsearch · GitHub
I hope this helps.