Not able to get fields from multiple source indices

[Sheereen (Sheereen Hamza)]

Hi,

I gave multiple indices in the 'indices' field of the enrich policy. But it is fetching values from one index only.

Thanks

[jsanz (Jorge Sanz)]

You need to provide more details to understand why your pipeline is not running. Sharing your actual pipeline would help.

I just tested a pipeline with an enrich processor defined with two indices and worked as expected. Sharing the whole test:

# Create a sample servers index and add some data
PUT /delete_enrich_servers
{
  "mappings": {
    "properties": {
      "name": { "type": "keyword"},
      "location": { "type": "keyword"},
      "brand": { "type": "keyword"}
    }
  }
}

POST delete_enrich_servers/_bulk
{ "index" : { } }
{ "name" : "s1", "location": "loc2", "brand": "b2" }
{ "index" : { } }
{ "name" : "s2", "location": "loc3", "brand": "b1" }
{ "index" : { } }
{ "name" : "s3", "location": "loc3", "brand": "b1" }
{ "index" : { } }
{ "name" : "s4", "location": "loc2", "brand": "b2" }


# Create a sample routers index and add some data
PUT /delete_enrich_routers
{
  "mappings": {
    "properties": {
      "name": { "type": "keyword"},
      "location": { "type": "keyword"},
      "brand": { "type": "keyword"}
    }
  }
}

POST delete_enrich_routers/_bulk
{ "index" : { } }
{ "name" : "r1", "location": "loc1", "brand": "b1" }
{ "index" : { } }
{ "name" : "r2", "location": "loc1", "brand": "b2" }
{ "index" : { } }
{ "name" : "r3", "location": "loc2", "brand": "b2" }
{ "index" : { } }
{ "name" : "r4", "location": "loc2", "brand": "b1" }


# Create an enrich policy to add location and brand given a name
PUT /_enrich/policy/delete-routers-servers
{
  "match": {
    "indices": ["delete_enrich_servers", "delete_enrich_routers"],
    "match_field": "name",
    "enrich_fields": ["location", "brand"]
  }
}

# Execute the policy
POST /_enrich/policy/delete-routers-servers/_execute


# Create an ingest pipeline with the enrich policy
PUT /_ingest/pipeline/delete-routers-servers-lookup
{
  "processors" : [
    {
      "enrich" : {
        "description": "Add 'location' and 'brand' data based on 'name'",
        "policy_name": "delete-routers-servers",
        "field" : "name",
        "target_field": "info",
        "max_matches": "1"
      }
    }
  ]
}

# Add a couple of documents using the pipeline
PUT delete-devices
{
  "mappings": {
    "properties": {
      "name": {"type": "keyword"}
    }
  }
}

POST delete-devices/_bulk?pipeline=delete-routers-servers-lookup
{ "index" : { } }
{ "name" : "r1" }
{ "index" : { } }
{ "name" : "s3" }

# Check the documents where enriched
GET delete-devices/_search


# Clean up
DELETE _ingest/pipeline/delete-routers-servers-lookup
DELETE _enrich/policy/delete-routers-servers
DELETE delete-devices
DELETE delete_enrich_routers
DELETE delete_enrich_servers

The result of GET delete-devices/_search shows the info field with the enriched data as expected.

Hope it helps.

[Sheereen (Sheereen Hamza)]

Thanks for the response.

GET student_info/_search
 "hits" : [
      {
          "id" : "HDSJ3636",
          "name" : "Anna",
          "age" : 8
      },
      {
          "id" : "HDSJ8936",
          "name" : "Maximus",
          "age" : 7
      },
      {
          "id" : "HDSJ3674",
          "name" : "Sarah",
          "age" : 10
      }
    ]

---

GET student_marks/_search

"hits" : [
      {
          "id" : "HDSJ3636",
          "maths" : 81,
          "english" : 74,
          "science" : 89
      },
      {
          "id" : "HDSJ8936",
          "maths" : 85,
          "english" : 64,
          "science" : 69
      },
      {
          "id" : "HDSJ37895",
          "maths" : 89,
          "english" : 89,
          "science" : 89
      }
    ]

PUT _enrich/policy/student_details_processor
{
  "match": {
    "indices": ["student_info", "student_marks"],
    "match_field": "id",
    "enrich_fields": ["name", "maths", "science"]
  }
}

POST _enrich/policy/student_details_processor/_execute

PUT _ingest/pipeline/student_details_pipeline
{
  "processors" : [
    {
      "enrich" : {
        "description": "Add 'name' and 'stem subject marks'",
        "policy_name": "student_details_processor",
        "field" : "id",
        "target_field": "info"
      }
    }
  ]
}

POST student_details/_doc?pipeline=student_details_pipeline
{
  "id": "HDSJ3636",
  "gender": "Female"
}

POST student_details/_doc?pipeline=student_details_pipeline
{
  "id": "HDSJ8936",
  "gender": "Male"
}

POST student_details/_doc?pipeline=student_details_pipeline
{
  "id": "HDSJ3674",
  "gender": "Female"
}

POST student_details/_doc?pipeline=student_details_pipeline
{
  "id": "HDSJ37895",
  "gender": "Male"
}

GET student_details/_search


 "hits" : [
      {
          "gender" : "Female",
          "id" : "HDSJ3636",
          "info" : {
            "name" : "Anna",
            "id" : "HDSJ3636"
          }
      },
      {
          "gender" : "Male",
          "id" : "HDSJ8936",
          "info" : {
            "name" : "Maximus",
            "id" : "HDSJ8936"
          }
      },
      {
          "gender" : "Female",
          "id" : "HDSJ3674",
          "info" : {
            "name" : "Sarah",
            "id" : "HDSJ3674"
          }
      },
      {
          "gender" : "Male",
          "id" : "HDSJ37895",
          "info" : {
            "maths" : 89,
            "science" : 89,
            "id" : "HDSJ37895"
          }
      }
    ]

[jsanz (Jorge Sanz)]

I see you want to get data from both indices. The enrich policy indices are meant (I suppose) to be complementary, meaning that they share a schema and your ingest documents should only get enriched by one document of any of the source indices.

For getting data from both I think that you need to set up two different policies, one per source index, and put them together as different processors in your pipeline.

[Sheereen (Sheereen Hamza)]

Oooh..K
What you have done is having data from both indices in your new index.... not combining data in the 2 indices.

I get it now...Thanks a lot