Hi all,
I was wondering if there it is possible to reduce the size of a scroll search dynamically.
I am using Elasticsearch 2.3. I am coding in Java if that helps.
My use case:
I am trying to move items from my ES cluster to another storage and one query can contain up to ~40million items. Long story short, I am running the compute/query on AWS Lambda and I am chaining a bunch of these Lambda functions passing the scrollId until the entire scroll query is complete. Pseudocode encompassing the idea:
if (scrollId is not null):
(records, scrollId) = scrollSearch(scrollId)
writeRecordsToOtherStorage(records)
startNextLambda(newScrollId)
The problem arises in that Lambda has a fixed time limit of 5 minutes. Since the records are not fixed length, In some cases, the task cannot complete in 5minutes. In these cases, when I am retrying, I want to be able to reduce the scroll size in half at whatever point I am in the search. For example, if I am starting a Search A with size = N, and after the 10th Lambda it fails, I want to be redo that Lambda with size N/2. The redo logic is all in place and fine, it is just that I cannot change the scroll size when restarting that Lambda.
I appreciate any feedback. Thank you in advance!