Not able to get fields from multiple source indices

Hi,

I gave multiple indices in the 'indices' field of the enrich policy. But it is fetching values from one index only.

Thanks

You need to provide more details to understand why your pipeline is not running. Sharing your actual pipeline would help.

I just tested a pipeline with an enrich processor defined with two indices and worked as expected. Sharing the whole test:

# Create a sample servers index and add some data
PUT /delete_enrich_servers
{
  "mappings": {
    "properties": {
      "name": { "type": "keyword"},
      "location": { "type": "keyword"},
      "brand": { "type": "keyword"}
    }
  }
}

POST delete_enrich_servers/_bulk
{ "index" : { } }
{ "name" : "s1", "location": "loc2", "brand": "b2" }
{ "index" : { } }
{ "name" : "s2", "location": "loc3", "brand": "b1" }
{ "index" : { } }
{ "name" : "s3", "location": "loc3", "brand": "b1" }
{ "index" : { } }
{ "name" : "s4", "location": "loc2", "brand": "b2" }


# Create a sample routers index and add some data
PUT /delete_enrich_routers
{
  "mappings": {
    "properties": {
      "name": { "type": "keyword"},
      "location": { "type": "keyword"},
      "brand": { "type": "keyword"}
    }
  }
}

POST delete_enrich_routers/_bulk
{ "index" : { } }
{ "name" : "r1", "location": "loc1", "brand": "b1" }
{ "index" : { } }
{ "name" : "r2", "location": "loc1", "brand": "b2" }
{ "index" : { } }
{ "name" : "r3", "location": "loc2", "brand": "b2" }
{ "index" : { } }
{ "name" : "r4", "location": "loc2", "brand": "b1" }


# Create an enrich policy to add location and brand given a name
PUT /_enrich/policy/delete-routers-servers
{
  "match": {
    "indices": ["delete_enrich_servers", "delete_enrich_routers"],
    "match_field": "name",
    "enrich_fields": ["location", "brand"]
  }
}

# Execute the policy
POST /_enrich/policy/delete-routers-servers/_execute


# Create an ingest pipeline with the enrich policy
PUT /_ingest/pipeline/delete-routers-servers-lookup
{
  "processors" : [
    {
      "enrich" : {
        "description": "Add 'location' and 'brand' data based on 'name'",
        "policy_name": "delete-routers-servers",
        "field" : "name",
        "target_field": "info",
        "max_matches": "1"
      }
    }
  ]
}

# Add a couple of documents using the pipeline
PUT delete-devices
{
  "mappings": {
    "properties": {
      "name": {"type": "keyword"}
    }
  }
}

POST delete-devices/_bulk?pipeline=delete-routers-servers-lookup
{ "index" : { } }
{ "name" : "r1" }
{ "index" : { } }
{ "name" : "s3" }

# Check the documents where enriched
GET delete-devices/_search


# Clean up
DELETE _ingest/pipeline/delete-routers-servers-lookup
DELETE _enrich/policy/delete-routers-servers
DELETE delete-devices
DELETE delete_enrich_routers
DELETE delete_enrich_servers

The result of GET delete-devices/_search shows the info field with the enriched data as expected.

Hope it helps.

1 Like

Thanks for the response.

GET student_info/_search
 "hits" : [
      {
          "id" : "HDSJ3636",
          "name" : "Anna",
          "age" : 8
      },
      {
          "id" : "HDSJ8936",
          "name" : "Maximus",
          "age" : 7
      },
      {
          "id" : "HDSJ3674",
          "name" : "Sarah",
          "age" : 10
      }
    ]

GET student_marks/_search

"hits" : [
      {
          "id" : "HDSJ3636",
          "maths" : 81,
          "english" : 74,
          "science" : 89
      },
      {
          "id" : "HDSJ8936",
          "maths" : 85,
          "english" : 64,
          "science" : 69
      },
      {
          "id" : "HDSJ37895",
          "maths" : 89,
          "english" : 89,
          "science" : 89
      }
    ]
PUT _enrich/policy/student_details_processor
{
  "match": {
    "indices": ["student_info", "student_marks"],
    "match_field": "id",
    "enrich_fields": ["name", "maths", "science"]
  }
}

POST _enrich/policy/student_details_processor/_execute

PUT _ingest/pipeline/student_details_pipeline
{
  "processors" : [
    {
      "enrich" : {
        "description": "Add 'name' and 'stem subject marks'",
        "policy_name": "student_details_processor",
        "field" : "id",
        "target_field": "info"
      }
    }
  ]
}
POST student_details/_doc?pipeline=student_details_pipeline
{
  "id": "HDSJ3636",
  "gender": "Female"
}

POST student_details/_doc?pipeline=student_details_pipeline
{
  "id": "HDSJ8936",
  "gender": "Male"
}

POST student_details/_doc?pipeline=student_details_pipeline
{
  "id": "HDSJ3674",
  "gender": "Female"
}

POST student_details/_doc?pipeline=student_details_pipeline
{
  "id": "HDSJ37895",
  "gender": "Male"
}
GET student_details/_search


 "hits" : [
      {
          "gender" : "Female",
          "id" : "HDSJ3636",
          "info" : {
            "name" : "Anna",
            "id" : "HDSJ3636"
          }
      },
      {
          "gender" : "Male",
          "id" : "HDSJ8936",
          "info" : {
            "name" : "Maximus",
            "id" : "HDSJ8936"
          }
      },
      {
          "gender" : "Female",
          "id" : "HDSJ3674",
          "info" : {
            "name" : "Sarah",
            "id" : "HDSJ3674"
          }
      },
      {
          "gender" : "Male",
          "id" : "HDSJ37895",
          "info" : {
            "maths" : 89,
            "science" : 89,
            "id" : "HDSJ37895"
          }
      }
    ]

I see you want to get data from both indices. The enrich policy indices are meant (I suppose) to be complementary, meaning that they share a schema and your ingest documents should only get enriched by one document of any of the source indices.

For getting data from both I think that you need to set up two different policies, one per source index, and put them together as different processors in your pipeline.

Oooh..K
What you have done is having data from both indices in your new index.... not combining data in the 2 indices.

I get it now...Thanks a lot

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.