I am creating a project that on the input has a large file that can have some flags that might be used in searching in elastic as well. Mostly the flags will be ones that will be changing. The only thing I know is, that the file has been changed but I do not know which part as I do not keep copy of the file. Also what I get is current version number of the file.
I expected that I will use version number as an external version for elastic.
Since most of the big data will be changed rarely while the flags will be changed often I would like to split the document stored in the elastic to multiple section where I would like to use parent-child relationship.
I planned to receive a document split it into bigger and smaller part and send it to the elastic as update with detect noops. the parts that would not change would not be reindexed.
However I have run into problem that update for some reason does not support external versioning. Seems little strange to me since I would expect if the version I would send to the elastic would be higher it would proceed with update and perform detect noop thing.
Sure I can use multi get to receive files from the elastic and check the source parts for changes, but that is not the same clean version as detect noops since there is an additional traffic and lets say if I use bulk command where one document would appear multiple times would not work at all.
For example lets say we would have indexed version 1.
Then two different nodes would try to send version 2 and version 3 at the same time. Version 3 would be same as version 1.
Both would get the document from the elastic and while version 2 would see changes needs to be done version 3 would not think so. In the end version 2 would be indexed in the elastic while version 3 left outside.
The ability to detect if the update is a noop one is much helpful and for these situations essential. Since we need to detect the if the change is a noop and commit like it would be atomic operation.
Is there some way how to use the external version with detect noops functionality? Also is there a reason why update API should refuse external version when the rule that the version must be increase with each change would remain?
Thank you for your answers.