Elastic search optimistic concurrency control not working as expected

[reddroid]

Hi,

We are using Elasticsearch version 7.16.2 with the Java Rest client to process and store documents in Elasticsearch in a Spring application .

We are noticing this intermittent issue where documents with lower seq no was saved when doing an Index request for an exiting document .

Implemetation :
First we do a find to check if the document exists . If exists , we do a index request and use the seq no from the get request to do the index request to update the document with the latest info.

Intermittently we are seeing this issue .

Chronology of Events:

1. Host 1 :
   Current Document seqno : 262738316 and primaryTerm:7 . Successfully indexed to Elasticsearch and this should have been the final update . We got back the new seq no as
   Seq no :262738317 and primaryTerm:7

2. Host 2:
   An older event was picked for processing with seqNo:262738315,primaryTerm:7 . This should not have not have been updated but Elasticsearch indexed this document anyways and the new seqno that was returned was seqNo:262738316,primaryTerm:7 . Which is wrong .

The time difference between the logs are 500ms. So , they are not happening at the same time.

Also, note that we have 3 servers that are in parallel processing and updating a ES cluster of 5 nodes.

Please has anyone noticed this issue ?

[DavidTurner]

It sounds like your host clocks are out of sync by more than 500ms.

[reddroid]

Not sure what you mean by that . Please can you elaborate ?

[DavidTurner]

Since everything is in the same primary term it means that the sequence numbers are being assigned by a single primary, which means they are assigned in order. So the update from 262738315 to 262738316 definitely took place before the update from 262738316 to 262738317. The usual explanation for seeing these things happen in the other order on different systems is that the timestamps on the other systems aren't accurate.