Elasticsearch scroll offset

Elastic Stack Elasticsearch
1

Hello, can I scroll with a predefined offset in Elasticsearch 2.x? For example, I want to scroll from offset 101 limit 100. We want to scroll over an enormous dataset for analytic purpose. Our JVM's heap size can't store over 100 million document at a time.

Since we can't estimate the time for a scroll, use a old scroll id after a network failure event will make our services become a mess. Is there any gracefully way to handle this?

I found this closed issue : Parallel/concurrent reads
But slice is not available in Elasticsearch 2.x.

Additional info: We'are using Java driver.

2

If you are on ES 2.x one way to achieve this is to pick a field where you can perform multiple ranges that return the same amount of documents. A date field for instance with whom you can slice your data by doing a range query per month/year/day depending on the repartition of you data within the date field. This way you can split your scroll query in smaller queries that you can parallelize.

3

This means you should use aggregations. Pulling the results back locally is almost always going to result in unacceptable performance even if you can get it working. You can totally write custom aggregations as plugins if you really need to.