Delete by query and date range causes unexpected "version_conflict_engine_exception", 409 response

Elastic Stack Elasticsearch
1

I am running a query to delete certain logs/entries before a certain date with a log level of "Debug" as shown here, notice the wildcard in the index name

POST /my_index_prod*/_delete_by_query
{
  "query": {
    "bool": {
      "must": [
        {
          "range": {
            "timestamp": {
                "lte": "now-30d/d"
            }
          }
        }
      ],
      "filter": [
        {
          "match_all": {}
        },
        {
          "match_phrase": {
            "level": "DEBUG"
          }
        }
      ],
      "should": [],
      "must_not": []
    }
  }
}

But i keep seeing that a lot of logs are catched by this condition but only a few deleted and the errors return include a lot of version_conflict_engine_exception

Example

{
  "took" : 39228,
  "timed_out" : false,
  "total" : 4594818,
  "deleted" : 2354,
  "batches" : 3,
  "version_conflicts" : 646,
  "noops" : 0,
  "retries" : {
    "bulk" : 0,
    "search" : 0
  },
  "throttled_millis" : 0,
  "requests_per_second" : -1.0,
  "throttled_until_millis" : 0,
  "failures" : [
    {
      "index" : "my_index_prod-filebeat-7.14.0-2022.05",
      "type" : "_doc",
      "id" : "lRZHBoEBMAL2EtIf6Pbg",
      "cause" : {
        "type" : "version_conflict_engine_exception",
        "reason" : "[lRZHBoEBMAL2EtIf6Pbg]: version conflict, required seqNo [2369859], primary term [1]. but no document was found",
        "index_uuid" : "gAa8nhj6Q2mt4mXnC_u9-w",
        "shard" : "0",
        "index" : "my_index_prod-filebeat-7.14.0-2022.05"
      },
      "status" : 409
    },
    {
      "index" : "my_index_prod-filebeat-7.14.0-2022.05",
      "type" : "_doc",
      "id" : "qBZHBoEBMAL2EtIf6Pbg",
      "cause" : {
        "type" : "version_conflict_engine_exception",
        "reason" : "[qBZHBoEBMAL2EtIf6Pbg]: version conflict, required seqNo [2369865], primary term [1]. but no document was found",
        "index_uuid" : "gAa8nhj6Q2mt4mXnC_u9-w",
        "shard" : "0",
        "index" : "my_index_prod-filebeat-7.14.0-2022.05"
      },
      "status" : 409
    },


...
  ]
}

What am I doing wrong and what can I do to fix this?

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.