Using Ingest node Enrich your data with "multiple" match fields

Hey there.

I've been experimenting with the newly added data enrichment funtions in ES 7.5 and came up with the following problem.

Let's pretend we have those indices:

article_to_group

{
  "article_key": 123456,
  "article_group": "foobars"
}

article_views_from_other_data_source

{
  "date": "2019-12-10",
  "article_group": "foobars",
  "other_views": 1440
}

Building the enrichtment process

I came up with the following so far:

PUT /_enrich/policy/group_policy
{
    "match": {
        "indices": [
            "article_to_group"
        ],
        "match_field": "article_key",
        "enrich_fields": ["article_group"]
    }
}
POST /_enrich/policy/group_policy/_execute

PUT /_enrich/policy/other_views_policy
{
    "match": {
        "indices": [
            "article_views_from_other_data_source"
        ],
        "match_field": "article_group",
        "enrich_fields": ["other_views"]
    }
}
POST /_enrich/policy/other_views_policy/_execute

// Create pipeline
PUT /_ingest/pipeline/views_lookup
{
  "description" : "Enriching...",
  "processors" : [
    {
        "enrich" : {
          "policy_name": "group_policy",
          "field" : "article_key",
          "target_field": "info",
          "max_matches": "1"
        }
    },
    {
      "enrich" : {
        "policy_name": "other_views_policy",
        "field" : "info.article_group",
        "target_field": "other_views",
        "max_matches": "1"
      }
    }
  ]
}

Execute our pipeline

POST /my_test_index/_doc/1?pipeline=views_lookup
{
  "article_key": 123456,
  "views": 1337
}

Result of GET /my_test_index/_doc/1 is

{
    "other_views" : {
      "other_views" : 1440,
      "article_group" : "foobars"
    },
    "article_key" : 123456,
    "views" : 1337,
    "info" : {
      "article_key" : 123456,
      "article_group" : "foobars"
    }
}

Unfortunately this won't work as soon as there are mutiple logs in the article_views_from_other_data_source with the same article_group, but different dates.

Is there a way to define date as a "second" match_field in the pipeline definition with the query field?

1 Like

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.