Hey folks, I have a quick question on how to handle a particular scenario I'm running into.
So we have a sync between a person's calendar events and Elasticsearch to index the events from Calendar and build some sort of reporting on top of it.
So in the setup, we track and index all the events within a window and the changes that come with it like event getting modified or removed, and we use the updated timestamp as the version conflict parameter.
Sometimes, we need to resync a particular user's events and to achieve this we use delete by query to remove that user's data from ES and reindex the new data. But since delete by query works off a snapshot, after delete by query runs, and we try again to reindex data we run into version conflict errors because of us using updated time as version control parameter but since delete by query ran on the same documents, it has incremented the earlier version causing us to run into conflicts.
i.e if there is an event that was updated at 12345 that we used as the version, and we ran delete by query on that document, it's version is now 12346 and when we reindex it causes version conflict between 12345 and 12346.
We've tried force merging and refreshing the index after the delete by query but still no avail.
Any ideas on how we can circumvent running into this issue?