We are using Elasticsearch version 7.16.2 with the Java Rest client to process and store documents in Elasticsearch in a Spring application .
We are noticing this intermittent issue where documents with lower seq no was saved when doing an Index request for an exiting document .
First we do a find to check if the document exists . If exists , we do a index request and use the seq no from the get request to do the index request to update the document with the latest info.
Intermittently we are seeing this issue .
Chronology of Events:
Host 1 :
Current Document seqno : 262738316 and primaryTerm:7 . Successfully indexed to Elasticsearch and this should have been the final update . We got back the new seq no as
Seq no :262738317 and primaryTerm:7
An older event was picked for processing with seqNo:262738315,primaryTerm:7 . This should not have not have been updated but Elasticsearch indexed this document anyways and the new seqno that was returned was seqNo:262738316,primaryTerm:7 . Which is wrong .
The time difference between the logs are 500ms. So , they are not happening at the same time.
Also, note that we have 3 servers that are in parallel processing and updating a ES cluster of 5 nodes.
Please has anyone noticed this issue ?