Thanks for the response! That makes sense, I agree that moving the refresh after insertion should resolve the problem.
My only open question is how come the 409 error occurs in the discussion here: Elasticsearch delete_by_query 409 version conflict
My expectation is that delete by query would just find no documents to delete since a refresh has not occurred since the documents were inserted. How can a 409 error occur if the delete by query only finds indexed documents? (I'm assuming by "indexed" you mean that a refresh has occurred after the insertion) It seems like from the discussion, the 409 error happens because somehow the delete by query finds an older version of the document that was just inserted. I'm probably missing something, but the document doesn't exist before the step of the document being indexed so how can there be a version conflict?
The only explanation I can think of is that the original poster meant that an update document request is sent instead of a create document request. Then delete by query would find the old version of the document and attempt to delete that and a 409 error will be returned