Hey there.
I've been experimenting with the newly added data enrichment funtions in ES 7.5 and came up with the following problem.
Let's pretend we have those indices:
article_to_group
{
"article_key": 123456,
"article_group": "foobars"
}
article_views_from_other_data_source
{
"date": "2019-12-10",
"article_group": "foobars",
"other_views": 1440
}
Building the enrichtment process
I came up with the following so far:
PUT /_enrich/policy/group_policy
{
"match": {
"indices": [
"article_to_group"
],
"match_field": "article_key",
"enrich_fields": ["article_group"]
}
}
POST /_enrich/policy/group_policy/_execute
PUT /_enrich/policy/other_views_policy
{
"match": {
"indices": [
"article_views_from_other_data_source"
],
"match_field": "article_group",
"enrich_fields": ["other_views"]
}
}
POST /_enrich/policy/other_views_policy/_execute
// Create pipeline
PUT /_ingest/pipeline/views_lookup
{
"description" : "Enriching...",
"processors" : [
{
"enrich" : {
"policy_name": "group_policy",
"field" : "article_key",
"target_field": "info",
"max_matches": "1"
}
},
{
"enrich" : {
"policy_name": "other_views_policy",
"field" : "info.article_group",
"target_field": "other_views",
"max_matches": "1"
}
}
]
}
Execute our pipeline
POST /my_test_index/_doc/1?pipeline=views_lookup
{
"article_key": 123456,
"views": 1337
}
Result of GET /my_test_index/_doc/1
is
{
"other_views" : {
"other_views" : 1440,
"article_group" : "foobars"
},
"article_key" : 123456,
"views" : 1337,
"info" : {
"article_key" : 123456,
"article_group" : "foobars"
}
}
Unfortunately this won't work as soon as there are mutiple logs in the article_views_from_other_data_source
with the same article_group
, but different dates.
Is there a way to define date as a "second" match_field
in the pipeline definition with the query field?