So I saw a question or two with regards to this topic but they were over a year old so I wanted to pop in and ask it again; hopefully something has changed.
My use case is this:
- I'm working in an event sourced, eventually consistent model where events are processed and used to issue partial updates/upserts to any data store. In this case, elastic search.
- The issue is that it is possible that events get picked up by different instances of the handler and processed out of order. As such, the solution has been to process every updated as an upsert.
- For the most part, this isn't an issue since every event has an Entity ID so it's just matter of upserting everything. The complication comes in a particular case where I'm actually tracking event store indexes, which is scalar in nature instead of it being an object. Each "projection" has only one index cache that contains one number which represents the last event index that was processed. This is used so that when my service restarts, it knows how to pick up where it left off.
The way that I have implemented it in MongoDB was to create a conditional upsert. In this case, the upsert only occurs if the value of a particular field in the document that I'm attempting to upsert (the index) is greater than the one that is currently there. This has to be accomplished in a single atomic transaction due to the concurrent nature of the denormalizer. That is, I can't query first and then upsert second in order to handle the condition myself because in between operations, the data will likely already become dirty.
How do I achieve this in ElasticSearch, or even better -- using NEST? It looked like the correct path to take is to use a script, but I saw that I cannot use scripts and documents at the same time, but that it has to be one or the other which makes managing the object a bit more difficult or messy at least. Is this still the case? That could get messy depending on the size of the upsert, which differs based on the event. For example PhoneNumberChangedprobably wouldn't be a problem but NewUserCreated would be significantly bigger. I saw that leveraging the document version could be a solution, but that feels really hacky... but maybe it's not, and I'm not sure how to do it or how it would play with this scenario. Any guidance is appreciated. At first glance at the documentation, it seems that it would only really work for non-partial operations...?