Elasticsearch enrich policy not reflecting updated source after _execute (v9.1.3)

Elastic Stack Elasticsearch
,
1

Hi, I’m running into an issue with enrich policies in Elasticsearch v9.1.3.

When I update the source index used by an enrich policy and then re-execute the policy, the enrich simulation still returns stale data. The updated fields are not visible right away.

Workarounds I found:

  • If I delete and recreate the policy, the updated values appear.
  • If I wait a while, the updates eventually show up.
  • If I delete and recreate the pipeline, it also works.

But ideally, the policy re-execution should pick up the changes immediately.

Repro Steps

1. Create the enrich index and seed some docs

PUT enrich_u_v1
{
  "mappings": {
    "properties": {
      "unit_ref": { "type": "keyword" },
      "id":       { "type": "keyword" },
      "name":     { "type": "keyword" },
      "ext_id":   { "type": "keyword" }
    }
  }
}

PUT enrich_u_v1/_doc/1?refresh=true
{"unit_ref":"U001","id":"1","name":"Unit One","ext_id":"A-1"}

PUT enrich_u_v1/_doc/2?refresh=true
{"unit_ref":"U002","id":"2","name":"Unit Two","ext_id":"A-2"}

2. Create the enrich policy and execute it

PUT _enrich/policy/u_policy
{
  "match": {
    "indices": "enrich_u_*",
    "match_field": "unit_ref",
    "enrich_fields": [ "id", "name", "ext_id" ]
  }
}

POST _enrich/policy/u_policy/_execute?wait_for_completion=true

3. Create a pipeline using the policy

PUT _ingest/pipeline/enrich_u_pipeline
{
  "processors": [
    {
      "enrich": {
        "override": true,
        "max_matches": 1,
        "policy_name": "u_policy",
        "field": "unit_ref",
        "target_field": "unit",
        "ignore_missing": true
      }
    }
  ]
}

4. Simulate with initial data

POST _ingest/pipeline/enrich_u_pipeline/_simulate
{
  "docs": [
    { "_source": { "unit_ref": "U001" } }
  ]
}

→ Works fine, enrich returns "Unit One".

5. Update the source index

POST enrich_u_v1/_update_by_query?refresh=true
{
  "script": {
    "source": "ctx._source.name = 'Unit One (UPDATED)'",
    "lang": "painless"
  },
  "query": { "term": { "unit_ref": "U001" } }
}

6. Re-execute the policy and simulate again

POST _enrich/policy/u_policy/_execute?wait_for_completion=true

POST _ingest/pipeline/enrich_u_pipeline/_simulate?verbose=true
{
  "docs": [
    { "_source": { "unit_ref": "U001" } }
  ]
}

→ Still shows "Unit One" instead of "Unit One (UPDATED)".

Expected Behavior

After updating the enrich index and re-executing the policy, I expect simulations to return the updated field values immediately.

Actual Behavior

Values stay stale until I:

  • wait some time, or
  • delete/recreate the policy or pipeline.

Question

Is this expected behavior (due to some caching/async refresh), or is there a missing step to force refresh the enrich index for the policy?

2

If I'm not wrong, this is not how it works, it may take some for an enrich policy execute to finish.

First It needs to create a new index with the documents from the source index, depending on the size of the source index and specs of the cluster this may take some time, after that it needs to force merge the index, which can also take some time, it is not something that runs immediatelly.

So, you need to wait some time for the data to be updated after an execute request.

3

Just figured out that from 9.1.0–9.1.3 there’s an issue where the enrich processor keeps using a cached view of the enrich index even after you re-execute the policy,

so simulations (and ingest) can return stale values. Elastic fixed this in 9.1.4

  • Fix enrich caches outdated value after policy run #133680

Upgrading to 9.1.4+ resolves it.

2 Likes