We have a spark application that needs to use
_delete_by_query to do some cleanup after it's finished indexing during a spark task. We're finding that this is causing us to hit the scroll context limit. Our delete requests tend to return very few results so should be executed quickly which leads me to wonder about the following
- Are the scroll contexts used by
_delete_by_querycleared after the delete is finished or left to expire naturally?
- Would a new scroll be created if the timeout was breached or would the entire request fail?