Use Enrich policy and enrich pipeline processor to check secondary index and update a value

Elastic Stack Elasticsearch
1

So i have 2 indexes, A which is periodically updated and contains docs with two fields which are email and reviewed. My second index B has lots of fields including email and reviewed, this index im constantly adding to.

I want a pipeline to check the incoming doc (incoming to index B) against Index A to see if it exists and if it does update the incoming doc "reviewed" boolean with the value in index A. Ive added my enrich policy and pipeline below, currently any matches do not get put into the index. Ive found that if i update the "target_field" in the pipeline to anything "test" for example it adds the incoming docs fine but instead of updating the value it puts test.user.email and test.user.reviewed into the doc.

I might be looking at this the wrong way so any suggestions as to how i should go about it would be appreciated.

index A
user.email_address, text
user.reviewed, boolean

index B
user.email_address, text
user.reviewed, boolean
...

This is my current pipeline

[
  {
    "enrich": {
     "policy_name": "reviewed-policy",
     "field": "user.email",
     "target_field": "user.reviewed}",
     "max_matches": "1",
      "on_failure": [
      {
        "set": {
         "field": "error.message",
          "value": "{{ _ingest.on_failure_message }}"
         }
        }
       ]
     }
 }
]

This is my current enrich policy

PUT /_enrich/policy/ignore-policy
{
 "match": {
 "indices": "indexA",
 "match_field": "user.email",
 "enrich_fields": ["reviewed"]
 }
}
2

So I have this working but its more of a work around rather than a fix, the enrich policy seems to only except a temporary field to save the user.email and user.reviewed. If you give it that such as the test example I used and follow the enrich processor with a set and remove processor to check for the temporary field with an if state and update/remove the value that way. Below is the code for that, hope this helps anyone.

[
  {
    "set": {
      "field": "user.reviewed",
      "value": false
    }
  },
  {
    "enrich": {
      "policy_name": "reviewed-policy",
      "field": "user.email",
      "target_field": "test",
      "max_matches": "1",
      "on_failure": [
        {
          "set": {
            "field": "error.message",
            "value": "{{ _ingest.on_failure_message }}"
          }
        }
      ]
    },
    "set": {
      "if": "ctx.containsKey('test')",
      "field": "user.reviewed",
      "value": "{{test.user.reviewed}}",
      "on_failure": [
        {
          "set": {
            "field": "error.message",
            "value": "Issue with user.reviewed set to {{test.user.reviewed}}"
          }
        }
      ]
    },
    "remove": {
      "if": "ctx.containsKey('test')",
      "field": [
        "test.user.email",
        "test.user.reviewed"
      ]
    }
  }
]
3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.