Add new documents to an enrich index

Hi, I have filebeat sending data to elasticsearch, this data is enriched with an ingest pipeline, I have added a new document to the enrich index that already is in use, but the changes are not showing.

What do I have to do so that these new documents are considered by the enrich processor?


Update an enrich policy

Once created, you can’t update or change an enrich policy. Instead, you can:

  1. Create and execute a new enrich policy.
  2. Replace the previous enrich policy with the new enrich policy in any in-use enrich processors.
  3. Use the delete enrich policy API to delete the previous enrich policy.

This may seem cumbersome, but it is by design, in essence it allows you "to version" your enrich policy. You can tag a date on the end...

PUT /_enrich/policy/my-policy-2022.11.11
  "match": {
    "indices": "users",
    "match_field": "email",
    "enrich_fields": ["first_name", "last_name", "city", "zip", "state"]

PUT /_enrich/policy/my-policy-2022.11.11/_execute

PUT /_ingest/pipeline/user_lookup
  "processors" : [
      "enrich" : {
        "description": "Add 'user' data based on 'email'",
        "policy_name": "my-policy-2022.11.11",
        "field" : "email",
        "target_field": "user",
        "max_matches": "1"

You can run a nightly script if you want...etc..

1 Like

As @stephenb said, you need to run the _execute request in your policy every time you update the source index of your enrich policy.

If you need to update your source indice frequently, you will need to schedule something to run this request, I have the same issue and I'm currently using a shell script on a crontab.

There is an open issue in github with a feature request to implement some way to schedule an execute on an enrich policy.

1 Like

Some reason I think that our the Threat Intel package does this (i could be wrong), it seems that if it does that functionality should be exposed for other enrich indices...

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.