# Ingest pipeline with update\_by\_query on failure causes version\_conflict\_engine\_exception

**URL:** <https://discuss.elastic.co/t/ingest-pipeline-with-update-by-query-on-failure-causes-version-conflict-engine-exception/266816>\
**Category:** Elasticsearch\
**Created:** [March 10, 2021, 1:57pm UTC](https://discuss.elastic.co/t/ingest-pipeline-with-update-by-query-on-failure-causes-version-conflict-engine-exception/266816 "2021-03-10T13:57:12Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![nuketro0p3r](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nuketro0p3r/32/32584_2.png) [@nuketro0p3r](https://discuss.elastic.co/u/nuketro0p3r)\
**Post date:** [March 10, 2021, 1:57pm UTC](https://discuss.elastic.co/t/ingest-pipeline-with-update-by-query-on-failure-causes-version-conflict-engine-exception/266816/1 "2021-03-10T13:57:12Z")

</div>

**Situation:**  
I set the "on\_failure" statement to an existing pipeline like follows (for the whole pipeline):

```auto
    "on_failure" : [
      {
        "set" : {
          "field" : "_index",
          "value" : "failed-{{ _index }}"
        }
      },
      {
        "set" : {
          "field" : "caused_by",
          "value" : "{{ _ingest.on_failure_message }}"
        }
      }
    ]

```

**Problem:**  
The pipeline is used with the update\_by\_query API to enrich documents. In case of a failure, the pipeline does not forward the failed document to the failed index ie. _failed-my\_index_. This results due to the following error message:

```auto
    {
      "took": 276,
      "timed_out": false,
      "total": 1,
      "updated": 0,
      "deleted": 0,
      "batches": 1,
      "version_conflicts": 1,
      "noops": 0,
      "retries": {
        "bulk": 0,
        "search": 0
      },
      "throttled_millis": 0,
      "requests_per_second": -1,
      "throttled_until_millis": 0,
      "failures": [
        {
          "index": "failed-my_index",
          "type": "_doc",
          "id": "1002109",
          "cause": {
            "type": "version_conflict_engine_exception",
            "reason": "[1002109]: version conflict, required seqNo [37265539], primary term [20]. but no document was found",
            "index_uuid": "JMepgCegQamU8WDqQuFJ3Q",
            "shard": "0",
            "index": "failed-my_index"
          },
          "status": 409
        }
      ]
    }

```

This is true, because the document "1002109" doesn't exist in the failed index, because it didn't fail when it was indexed originally. **On update, the pipeline fails, and the document needs to go into the failed index. This is not happening because of the version conflict error above.**

What am I doing wrong here?

Any help is appreciated

---

<div class="post-metadata">

**Author:** ![nuketro0p3r](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nuketro0p3r/32/32584_2.png) [@nuketro0p3r](https://discuss.elastic.co/u/nuketro0p3r)\
**Post date:** [March 24, 2021, 1:19pm UTC](https://discuss.elastic.co/t/ingest-pipeline-with-update-by-query-on-failure-causes-version-conflict-engine-exception/266816/2 "2021-03-24T13:19:32Z")

</div>

anyone?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 21, 2021, 1:19pm UTC](https://discuss.elastic.co/t/ingest-pipeline-with-update-by-query-on-failure-causes-version-conflict-engine-exception/266816/3 "2021-04-21T13:19:34Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.