Ingest pipeline with update_by_query on failure causes version_conflict_engine_exception

Elastic Stack Elasticsearch
1

Situation:
I set the "on_failure" statement to an existing pipeline like follows (for the whole pipeline):

    "on_failure" : [
      {
        "set" : {
          "field" : "_index",
          "value" : "failed-{{ _index }}"
        }
      },
      {
        "set" : {
          "field" : "caused_by",
          "value" : "{{ _ingest.on_failure_message }}"
        }
      }
    ]

Problem:
The pipeline is used with the update_by_query API to enrich documents. In case of a failure, the pipeline does not forward the failed document to the failed index ie. failed-my_index. This results due to the following error message:

    {
      "took": 276,
      "timed_out": false,
      "total": 1,
      "updated": 0,
      "deleted": 0,
      "batches": 1,
      "version_conflicts": 1,
      "noops": 0,
      "retries": {
        "bulk": 0,
        "search": 0
      },
      "throttled_millis": 0,
      "requests_per_second": -1,
      "throttled_until_millis": 0,
      "failures": [
        {
          "index": "failed-my_index",
          "type": "_doc",
          "id": "1002109",
          "cause": {
            "type": "version_conflict_engine_exception",
            "reason": "[1002109]: version conflict, required seqNo [37265539], primary term [20]. but no document was found",
            "index_uuid": "JMepgCegQamU8WDqQuFJ3Q",
            "shard": "0",
            "index": "failed-my_index"
          },
          "status": 409
        }
      ]
    }

This is true, because the document "1002109" doesn't exist in the failed index, because it didn't fail when it was indexed originally. On update, the pipeline fails, and the document needs to go into the failed index. This is not happening because of the version conflict error above.

What am I doing wrong here?

Any help is appreciated

2

anyone?

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.