Is there some way to wait for all outstanding indexing operations to complete, across the entire index? the refresh=wait_for parameter will only wait for the documents which are modified in this query, and only on the shards written to by this query...I need to wait for all writes index-wide.
Here's what I'm doing. I'm syncing a large data set, it takes many hours. I take a timestamp before I start my sync...then I spin through all the records from the primary store inserting them into elasticsearch writing each record with this timestamp. On completion I issue a delete_by_query to delete any records where the timestamp is less than the timestamp for this sync (to remove any old records which are no longer present in the primary store). This delete_by_query is very often failing with conflicts. I could issue all of my bulk requests with refresh=wait_for, but this makes the whole operation take several hours longer...I'd rather not do that. If I just issue the final call with refresh=wait_for it only applies to records (and shards) modified in that call specifically. Is there any way I can just wait for all outstanding writes, index-wide, before issuing my delete_by_query?
It's possible simply retrying the delete_by_query until it succeeds would actually do what I want...but what happens if the delete_by_query executes and succeeds on a document which has an outstanding index request? Will the index request re-insert the document? I kinda doubt it...at the very least it seems like a sketchy behavior to be relying on to guarantee consistent data. Any insight on this is greatly appreciated