Add new documents to an enrich index

[ElasticLiver (Dense Man)]

Hi, I have filebeat sending data to elasticsearch, this data is enriched with an ingest pipeline, I have added a new document to the enrich index that already is in use, but the changes are not showing.

What do I have to do so that these new documents are considered by the enrich processor?

Thanks!

[stephenb (Stephen Brown)]

Set up an enrich processor | Elasticsearch Guide [8.11] | Elastic

Update an enrich policy

Once created, you can’t update or change an enrich policy. Instead, you can:

1. Create and execute a new enrich policy.
2. Replace the previous enrich policy with the new enrich policy in any in-use enrich processors.
3. Use the delete enrich policy API to delete the previous enrich policy.

This may seem cumbersome, but it is by design, in essence it allows you "to version" your enrich policy. You can tag a date on the end...

PUT /_enrich/policy/my-policy-2022.11.11
{
  "match": {
    "indices": "users",
    "match_field": "email",
    "enrich_fields": ["first_name", "last_name", "city", "zip", "state"]
  }
}

PUT /_enrich/policy/my-policy-2022.11.11/_execute


PUT /_ingest/pipeline/user_lookup
{
  "processors" : [
    {
      "enrich" : {
        "description": "Add 'user' data based on 'email'",
        "policy_name": "my-policy-2022.11.11",
        "field" : "email",
        "target_field": "user",
        "max_matches": "1"
      }
    }
  ]
}

You can run a nightly script if you want...etc..

[leandrojmp (Leandro Pereira)]

@ElasticLiver

What do I have to do so that these new documents are considered by the enrich processor?

As @stephenb said, you need to run the _execute request in your policy every time you update the source index of your enrich policy.

If you need to update your source indice frequently, you will need to schedule something to run this request, I have the same issue and I'm currently using a shell script on a crontab.

There is an open issue in github with a feature request to implement some way to schedule an execute on an enrich policy.

[stephenb (Stephen Brown)]

Some reason I think that our the Threat Intel package does this (i could be wrong), it seems that if it does that functionality should be exposed for other enrich indices...