# Embedding generation using E5 failing for some records

**URL:** <https://discuss.elastic.co/t/embedding-generation-using-e5-failing-for-some-records/374090>\
**Category:** Elasticsearch\
**Tags:** elastic-stack-machine-learning\
**Created:** [February 4, 2025, 9:57pm UTC](https://discuss.elastic.co/t/embedding-generation-using-e5-failing-for-some-records/374090 "2025-02-04T21:57:05Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![Carlos\_Fernando\_Palm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/carlos_fernando_palm/32/77804_2.png) [@Carlos\_Fernando\_Palm](https://discuss.elastic.co/u/Carlos_Fernando_Palm)\
**Post date:** [February 4, 2025, 9:57pm UTC](https://discuss.elastic.co/t/embedding-generation-using-e5-failing-for-some-records/374090/1 "2025-02-04T21:57:05Z")

</div>

I am trying to use the E5 model to generate embeddings for some documents.

I used a reindex to generate the embeddings:

```auto
POST _reindex?wait_for_completion=false
{
  "source": {
    "index": "source_index",
    "size": 50 
  },
  "dest": {
    "index": "destination_index",
    "pipeline": "e5-test"
  }
}

```

After almost 8 minutes I got this error:

```auto
"response": {
    "took": 472742,
    "timed_out": false,
    "total": 1997,
    "updated": 1749,
    "created": 0,
    "deleted": 0,
    "batches": 35,
    "version_conflicts": 0,
    "noops": 0,
    "retries": {
      "bulk": 0,
      "search": 0
    },
    "throttled": "0s",
    "throttled_millis": 0,
    "requests_per_second": -1,
    "throttled_until": "0s",
    "throttled_until_millis": 0,
    "failures": [
      {
        "index": "destination_index",
        "id": "324723984",
        "cause": {
          "type": "array_index_out_of_bounds_exception",
          "reason": "Index 9 out of bounds for length 8"
        },
        "status": 500
      }
    ]
  }

```

I see that none of the embeddings were generated, apparently because of this single record that failed.  
Is there a way for reindex to continue when a single document fails the embedding generation process?

---

<div class="post-metadata">

**Author:** ![iulia](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/iulia/32/124658_2.png) [@iulia](https://discuss.elastic.co/u/iulia)\
**Post date:** [February 20, 2025, 9:45am UTC](https://discuss.elastic.co/t/embedding-generation-using-e5-failing-for-some-records/374090/2 "2025-02-20T09:45:12Z")

</div>

Hi!

You can enable `error handling` on an re-index pipeline (in this case your `e5-test`) and decide what happens if an error occurs. Otherwise, by default the pipeline will indeed stop when it encounters an error.

You can see the [docs here](https://www.elastic.co/guide/en/elasticsearch/reference/current/ingest.html#handling-pipeline-failures)

You can either set `"ignore_failure": True` to completely ignore those data points; or even better set a message explaining the issue:

```auto
"on_failure": [
      {
           "set": {
                "description": "Set 'error.message'",
                "field": "error.message",
                "value": "Field 'provider' does not exist. Cannot rename to 'cloud.provider'",
                "override": False
              }
      }
]

```

If you want the pipeline to continue after an error, specify these properties within the `processor` field of the pipeline (not within the pipeline properties); otherwise the pipeline will still stop after running into an issue.

Hope this helps!