Version Conflict Engine Exception - seqNo question

[Anabella_Cristaldi (Anna)]

Hi All,
I'm getting version_conflict_engine_exception when doing an update by query in an index with one shard and no replicas.
I do not understand well why is this situation happening.
Is there any place in the doc where it is explained the conditions under this exception is raised?

The update by query I'm doing is:

POST audit-trail-2021.05.07-000001/_update_by_query
{
 "query": {
  "range": {
    "@timestamp": {
      "lte": "now-1h"
    }
  }
}, 
  "script": {
    "source": """ctx._source.source = [ "ip": ctx._source.ip ]""",
    "lang": "painless"
  }
}

and the error

  {
          "index" : "audit-trail-2021.05.07-000001",
          "type" : "_doc",
          "id" : "agzwWHkBtct8ebcgWHR6",
          "cause" : {
            "type" : "version_conflict_engine_exception",
            "reason" : "[agzwWHkBtct8ebcgWHR6]: version conflict, required seqNo [13740177], primary term [1]. current document has seqNo [19475784] and primary term [1]",
            "index_uuid" : "1wZ7iI_CS760nKU2sKgVEA",
            "shard" : "0",
            "index" : "audit-trail-2021.05.07-000001"
          },
          "status" : 409
        }

If I run the update by query with ?conflicts=proceed it executes well, but I want to understand the nature of the error
Thank you very much in advance
Regards
Ana

[Anabella_Cristaldi (Anna)]

I suppose that it is related to [this]
(Optimistic concurrency control | Elasticsearch Guide [7.12] | Elastic)

So I run the following experiment

1. In the scope of the documents I want to update I wanted to know the max seq_no, so I've executed this

    GET audit-trail/_search?version=true
    {
    "seq_no_primary_term": true,
    "track_total_hits": true, 
    "query": {
      "bool": { 
        "must": [
          {"wildcard": {"ip": {"value": "192*"}}},
          {"range": {
            "@timestamp": {
              "lte": "now-1h"
            }
          }}
        ]
      }
    }, "sort": [
      {
        "_seq_no": {
          "order": "desc"
        }
      }
    ]
    }
   

and the document with highest seqNo is 37250895

2. After that I perform the update by query

    POST audit-trail-2021.05.07-000001/_update_by_query
    {
    "query": {
      "bool": { 
        "must": [
          {"wildcard": {"ip": {"value": "192*"}}},
          {"range": {
            "@timestamp": {
              "lte": "now-1h"
            }
          }}
        ]
      }
    }, 
      "script": {
        "source": """ctx._source.source = [ "ip": ctx._source.ip ]""",
        "lang": "painless"
      }
    }
   

I got the version_conflict_engine_exception

How the required seqNo for this new update operation is lower than the max seqNo of the existing documents? (documents once indexed are not modified)
How the required seqNo for the update by query operation is determined?

Probably I'm missing something...

Thank you!
Regards
Ana