Can't use multiple enrich policies in the same ingest pipeline

Elastic Stack Elasticsearch
1

Hi Im trying to add enrich policies in the same ingest pipeline, but when I try to test it with this simulation I get an error.

POST /_ingest/pipeline/my-pipe/_simulate?verbose=true
{
  "docs": [
    {
      "_index": "index",
      "_id": "id",
      "_source": {
        "ServiceCode": "LHO0000005"
      }
    }
  ]
}

This is the error I get:

{
  "docs": [
    {
      "processor_results": [
        {
          "processor_type": "enrich",
          "status": "success",
          "doc": {
            "_id": "id",
            "_index": "index",
            "_version": "-3",
            "_source": {
              "new": {
                "codigo_servicio": "LHO0000005",
                "servicio": "Cancel LH to Paytech"
              },
              "ServiceCode": "LHO0000005"
            },
            "_ingest": {
              "pipeline": "my-pipe",
              "timestamp": "2022-10-21T16:16:47.294338399Z"
            }
          }
        },
        {
          "processor_type": "enrich",
          "status": "error",
          "error": {
            "root_cause": [
              {
                "type": "illegal_argument_exception",
                "reason": "field [Syscode] not present as part of path [Syscode]"
              }
            ],
            "type": "illegal_argument_exception",
            "reason": "field [Syscode] not present as part of path [Syscode]"
          }
        }
      ]
    }
  ]
}

This is my ingest pipeline:

{
  "my-pipe": {
    "processors": [
      {
        "enrich": {
          "field": "ServiceCode",
          "policy_name": "agrega_servicios",
          "target_field": "new"
        }
      },
      {
        "enrich": {
          "field": "Syscode",
          "policy_name": "agrega_sistemas",
          "target_field": "new"
        }
      },
      {
        "enrich": {
          "field": "ProcessCode",
          "policy_name": "agrega_productos",
          "target_field": "new"
        }
      }
    ]
  }
}

So, when its get to the second enrich policy produce an error. it all about the order of the procesors. they all work alone, but not when I use them in the same pipeline

Any advice?

2

Please share your enrich policies.

1 Like
3 

PUT /_enrich/policy/agrega_productos
{
  "match": {
    "indices": "diccionario_productos",  
    "match_field": "codigo_proceso",   
    "enrich_fields": ["producto"]       
  }
}

PUT /_enrich/policy/agrega_servicios
{
  "match": {
    "indices": "diccionario_servicios",   
    "match_field": "codigo_servicio",     
    "enrich_fields": ["servicio"]      
  }
}

PUT /_enrich/policy/agrega_sistemas
{
  "match": {
    "indices": "diccionario_sistemas",   
    "match_field": "codigo_sistema",  
    "enrich_fields": ["sistema"]     
}
4

Ohh Edit... Sorry

POST /_ingest/pipeline/my-pipe/_simulate?verbose=true
{
  "docs": [
    {
      "_index": "index",
      "_id": "id",
      "_source": {
        "ServiceCode": "LHO0000005"
      }
    }
  ]
}

If this ^^^ is the document you are testing of course it will fail.

There is no field Syscode
"reason": "field [Syscode] not present as part of path [Syscode]"

If you want to to drop through you need to add the

ignore_failure: true to each processor

Your pipeline as you defined would only work if you had all 3 fields

ServiceCode
Syscode
ProcessCode
2 Likes
5

Thanks @stephenb! I also realize that if I add a condition can make it work too ctx.containsKey('<source field')

1 Like
6

Oh yes conditions... that is good way too!

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.