Hi,
I gave multiple indices in the 'indices' field of the enrich policy. But it is fetching values from one index only.
Thanks
Hi,
I gave multiple indices in the 'indices' field of the enrich policy. But it is fetching values from one index only.
Thanks
You need to provide more details to understand why your pipeline is not running. Sharing your actual pipeline would help.
I just tested a pipeline with an enrich processor defined with two indices and worked as expected. Sharing the whole test:
# Create a sample servers index and add some data
PUT /delete_enrich_servers
{
"mappings": {
"properties": {
"name": { "type": "keyword"},
"location": { "type": "keyword"},
"brand": { "type": "keyword"}
}
}
}
POST delete_enrich_servers/_bulk
{ "index" : { } }
{ "name" : "s1", "location": "loc2", "brand": "b2" }
{ "index" : { } }
{ "name" : "s2", "location": "loc3", "brand": "b1" }
{ "index" : { } }
{ "name" : "s3", "location": "loc3", "brand": "b1" }
{ "index" : { } }
{ "name" : "s4", "location": "loc2", "brand": "b2" }
# Create a sample routers index and add some data
PUT /delete_enrich_routers
{
"mappings": {
"properties": {
"name": { "type": "keyword"},
"location": { "type": "keyword"},
"brand": { "type": "keyword"}
}
}
}
POST delete_enrich_routers/_bulk
{ "index" : { } }
{ "name" : "r1", "location": "loc1", "brand": "b1" }
{ "index" : { } }
{ "name" : "r2", "location": "loc1", "brand": "b2" }
{ "index" : { } }
{ "name" : "r3", "location": "loc2", "brand": "b2" }
{ "index" : { } }
{ "name" : "r4", "location": "loc2", "brand": "b1" }
# Create an enrich policy to add location and brand given a name
PUT /_enrich/policy/delete-routers-servers
{
"match": {
"indices": ["delete_enrich_servers", "delete_enrich_routers"],
"match_field": "name",
"enrich_fields": ["location", "brand"]
}
}
# Execute the policy
POST /_enrich/policy/delete-routers-servers/_execute
# Create an ingest pipeline with the enrich policy
PUT /_ingest/pipeline/delete-routers-servers-lookup
{
"processors" : [
{
"enrich" : {
"description": "Add 'location' and 'brand' data based on 'name'",
"policy_name": "delete-routers-servers",
"field" : "name",
"target_field": "info",
"max_matches": "1"
}
}
]
}
# Add a couple of documents using the pipeline
PUT delete-devices
{
"mappings": {
"properties": {
"name": {"type": "keyword"}
}
}
}
POST delete-devices/_bulk?pipeline=delete-routers-servers-lookup
{ "index" : { } }
{ "name" : "r1" }
{ "index" : { } }
{ "name" : "s3" }
# Check the documents where enriched
GET delete-devices/_search
# Clean up
DELETE _ingest/pipeline/delete-routers-servers-lookup
DELETE _enrich/policy/delete-routers-servers
DELETE delete-devices
DELETE delete_enrich_routers
DELETE delete_enrich_servers
The result of GET delete-devices/_search
shows the info
field with the enriched data as expected.
Hope it helps.
Thanks for the response.
GET student_info/_search
"hits" : [
{
"id" : "HDSJ3636",
"name" : "Anna",
"age" : 8
},
{
"id" : "HDSJ8936",
"name" : "Maximus",
"age" : 7
},
{
"id" : "HDSJ3674",
"name" : "Sarah",
"age" : 10
}
]
GET student_marks/_search
"hits" : [
{
"id" : "HDSJ3636",
"maths" : 81,
"english" : 74,
"science" : 89
},
{
"id" : "HDSJ8936",
"maths" : 85,
"english" : 64,
"science" : 69
},
{
"id" : "HDSJ37895",
"maths" : 89,
"english" : 89,
"science" : 89
}
]
PUT _enrich/policy/student_details_processor
{
"match": {
"indices": ["student_info", "student_marks"],
"match_field": "id",
"enrich_fields": ["name", "maths", "science"]
}
}
POST _enrich/policy/student_details_processor/_execute
PUT _ingest/pipeline/student_details_pipeline
{
"processors" : [
{
"enrich" : {
"description": "Add 'name' and 'stem subject marks'",
"policy_name": "student_details_processor",
"field" : "id",
"target_field": "info"
}
}
]
}
POST student_details/_doc?pipeline=student_details_pipeline
{
"id": "HDSJ3636",
"gender": "Female"
}
POST student_details/_doc?pipeline=student_details_pipeline
{
"id": "HDSJ8936",
"gender": "Male"
}
POST student_details/_doc?pipeline=student_details_pipeline
{
"id": "HDSJ3674",
"gender": "Female"
}
POST student_details/_doc?pipeline=student_details_pipeline
{
"id": "HDSJ37895",
"gender": "Male"
}
GET student_details/_search
"hits" : [
{
"gender" : "Female",
"id" : "HDSJ3636",
"info" : {
"name" : "Anna",
"id" : "HDSJ3636"
}
},
{
"gender" : "Male",
"id" : "HDSJ8936",
"info" : {
"name" : "Maximus",
"id" : "HDSJ8936"
}
},
{
"gender" : "Female",
"id" : "HDSJ3674",
"info" : {
"name" : "Sarah",
"id" : "HDSJ3674"
}
},
{
"gender" : "Male",
"id" : "HDSJ37895",
"info" : {
"maths" : 89,
"science" : 89,
"id" : "HDSJ37895"
}
}
]
I see you want to get data from both indices. The enrich policy indices are meant (I suppose) to be complementary, meaning that they share a schema and your ingest documents should only get enriched by one document of any of the source indices.
For getting data from both I think that you need to set up two different policies, one per source index, and put them together as different processors in your pipeline.
Oooh..K
What you have done is having data from both indices in your new index.... not combining data in the 2 indices.
I get it now...Thanks a lot
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
© 2020. All Rights Reserved - Elasticsearch
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant logo are trademarks of the Apache Software Foundation in the United States and/or other countries.