Elasticsearch | Relationships based index

Nikesh · April 9, 2019, 3:12pm

As asked in this link
I need to provide relationship based search.
Let's consider the same scenario.
Team and Players would contain similar fields.
A single team can have multiple players.
Having a Single Index for this would be something of this structure :

 {
 "Team_name" : "xyz",
 "type" : "men",
"Players" : [
       {
      "first" : "John",
      "last" :  "Smith",
      "country_ID" : "123"
    },
    {
      "first" : "Alice",
      "last" :  "Smith",
      "country_ID" : "679"
    }
]

}

I have few doubts over this architecture :

Any update to a particular part of "Players" array. Let's say " first" : "John" is changed to "first":"Mark"
This updating procedure requires reindexing the entire nested document.
Is there a procedure to Update just the array element to which it belongs to?
I would be using copy_to field and search would be based on accessibility and this is done through "country_ID". If a user has access to "Country_ID" :"679" and not to "123".
If a user searches for a string "Smith", will "Smith" with "country_ID" :"123"
also be searched and highlighted? If so, how do we prevent it from happening?
Suppose one player has to be deleted, let's say Alice Smith , Is there a procedure to just delete this player or does it require reindexing of this document without details of Alice Smith?

Nikesh · April 9, 2019, 3:13pm

@dadoonet Apologies for creating new thread as link has been closed.
Please provide any suggestions

dadoonet · April 9, 2019, 4:10pm

No
Not sure I understood but it's easy with Elasticsearch and Kibana dev console to test it.
You need to reindex the whole document.

Nikesh · April 10, 2019, 10:46am

My mapping is defined such that there exists a copy_to field enabled. This means Nested Object "Players" and all the other fields part of this JSON are also part of copy_to field.
Users might have access to search on only few players. For example, I have access to search only on "John Smith" and not "Alice Smith". This accessibility is based on "country_ID", which is part of my query through filter. I am using the "copy_to" field to search.
Incase I search on "Smith", I have to make sure it searches only "Smith" with "country_ID" :"123" and not "country_ID" : "679 and only the former should be part of the result.

dadoonet · April 10, 2019, 11:47am

Could you provide a full recreation script as described in About the Elasticsearch category. It will help to better understand what you are doing. Please, try to keep the example as simple as possible.

A full reproduction script will help readers to understand, reproduce and if needed fix your problem. It will also most likely help to get a faster answer.

Nikesh · April 10, 2019, 2:49pm

Thanks for the response.
Sure!

This is my mapping :

{
  "mappings": {
    "_doc": {
      "properties": {
      	"SINGLE_FIELD": {
                        "type": "text"
                        
                    },
      	"team_name" : {
      		"type": "text",
      		"copy_to": ["SINGLE_FIELD"]
      		
      	},
      "type" : {
      "type": "text",
     "copy_to": ["SINGLE_FIELD"]
      },
        "players": {
          "type": "nested",
          "properties": {
             "first" : {
        			"type" : "text",
        			"copy_to": ["SINGLE_FIELD"]
             },
             "last":{
             	"type" : "text",
             	"copy_to": ["SINGLE_FIELD"]
             },
             "country_ID" :{
             	"type" : "text",
             	"copy_to": ["SINGLE_FIELD"]
             }
          }
        }
        
      }
    }
  }
}

And I have indexed one document :

{
 "team_name" : "xyz",
 "type" : "men",
"Players" : [
       {
      "first" : "John",
      "last" :  "Smith",
      "country_ID" : "123"
    },
    {
      "first" : "Alice",
      "last" :  "Smith",
      "country_ID" : "679"
    }
]

}

My query is :

{
	"query": {

         
		"multi_match": {
            "query":"Smith",
            
            "fields":["SINGLE_FIELD"]
			
		}

	}
}

Now if I as a user has access to only player with country_ID : 679 and not to 123.
How do I restrict my search and highlight on country_ID : 123?

dadoonet · April 11, 2019, 2:28pm

Use a bool query.
Add the multi_match within the must clause and add a term query on country_ID field within the filter clause.

Nikesh · April 12, 2019, 4:25am

thanks for your response.

You meant query structure of the below sort?

{
  "query": {
    "bool": {
      "must": [
        {
          "multi_match": {
            "query": "Smith",
            "fields": [
              "SINGLE_FIELD"
            ]
          }
        }
      ],
      "filter": [
        {
          "term": {
            "country_ID": {
              "value": "679"
            }
          }
        }
      ]
    }
  }
}

dadoonet · April 12, 2019, 4:44am

Yes.

Nikesh · April 12, 2019, 12:18pm

Thanks for the response.
But this is not providing me any output.
Although the below query without any filter fetches me results -

{
  "query": {
    "bool": {
      "must": [
        {
          "multi_match": {
            "query": "Smith",
            "fields": [
              "SINGLE_FIELD"
            ]
          }
        }
      ]
    }
  }
}

Is something wrong with my filter clause?
this is my filter clause -

  "filter": [
    {
      "term": {
        "country_ID": {
          "value": "679"
        }
      }
    }
  ]

dadoonet · April 12, 2019, 12:41pm

Could you provide a full recreation script as described in About the Elasticsearch category. It will help to better understand what you are doing. Please, try to keep the example as simple as possible.

A full reproduction script will help readers to understand, reproduce and if needed fix your problem. It will also most likely help to get a faster answer.

Nikesh · April 15, 2019, 7:31am

Yes. The below is my query. This does not output any result.

   {
	"query": {

          "bool": {"must": [   {
		"multi_match": {
            "query":"Smith",
            
            "fields":["SINGLE_FIELD"]
			
		}
          }
          ],
          
            "filter": [
    {
      "term": {
        "players.country_ID": {
          "value": "679"
        }
      }
    }
  ]
          
          
          }

	}
}

dadoonet · April 15, 2019, 8:27am

There's no way to reproduce the problem without a full script. Unless someone wants to assemble it from the snippets you provided.
I'd recommend providing it yourself as described per the link I shared.

Nikesh · April 15, 2019, 10:31am

This is my mapping :

{
  "mappings": {
    "_doc": {
      "properties": {
      	"SINGLE_FIELD": {
                        "type": "text"
                        
                    },
      	"team_name" : {
      		"type": "text",
      		"copy_to": ["SINGLE_FIELD"]
      		
      	},
      "type" : {
      "type": "text",
     "copy_to": ["SINGLE_FIELD"]
      },
        "players": {
          "type": "nested",
          "properties": {
             "first" : {
        			"type" : "text",
        			"copy_to": ["SINGLE_FIELD"]
             },
             "last":{
             	"type" : "text",
             	"copy_to": ["SINGLE_FIELD"]
             },
             "country_ID" :{
             	"type" : "text",
             	"copy_to": ["SINGLE_FIELD"]
             }
          }
        }
        
      }
    }
  }
}

And I have indexed one document :

{
 "team_name" : "xyz",
 "type" : "men",
"Players" : [
       {
      "first" : "John",
      "last" :  "Smith",
      "country_ID" : "123"
    },
    {
      "first" : "Alice",
      "last" :  "Smith",
      "country_ID" : "679"
    }
]

}

The below is my query. This does not output any result.

   {
	"query": {

          "bool": {"must": [   {
		"multi_match": {
            "query":"Smith",
            
            "fields":["SINGLE_FIELD"]
			
		}
          }
          ],
          
            "filter": [
    {
      "term": {
        "players.country_ID": {
          "value": "679"
        }
      }
    }
  ]
          
          
          }

	}
}

Now if I as a user has access to only player with country_ID : 679 and not to 123 .
How do I restrict my search and highlight on country_ID : 123 ?

Nikesh · April 17, 2019, 6:44am

@dadoonet Please provide your suggestion

dadoonet · April 17, 2019, 2:40pm

@Nikesh let me paste the documentation I linked to in case you did not read it yet.

A typical script like this one can be copied and pasted in Kibana Dev Console by any reader. It will definitely help to play with your example and provide a fix for your script.
If you don't provide it, there is a chance that nobody can help.

Emphasis is mine.

Here is a typical script that anyone can use for example:

DELETE index
PUT index/_doc/1
{
  "foo": "bar"
}
GET index/_search
{
  "query": {
    "match": {
      "foo": "bar"
    }
  }
}

So if you need more help, please provide this.

Nikesh · April 19, 2019, 7:54am

Okay!

**PUT** nestedindex

    {
      "mappings": {
        "_doc": {
          "properties": {
          	"SINGLE_FIELD": {
                            "type": "text"
                            
                        },
          	"team_name" : {
          		"type": "text",
          		"copy_to": ["SINGLE_FIELD"]
          		
          	},
          "type" : {
          "type": "text",
         "copy_to": ["SINGLE_FIELD"]
          },
            "players": {
              "type": "nested",
              "properties": {
                 "first" : {
            			"type" : "text",
            			"copy_to": ["SINGLE_FIELD"]
                 },
                 "last":{
                 	"type" : "text",
                 	"copy_to": ["SINGLE_FIELD"]
                 },
                 "country_ID" :{
                 	"type" : "text",
                 	"copy_to": ["SINGLE_FIELD"]
                 }
              }
            }
            
          }
        }
      }
    }

**POST** nestedindex/_doc/1

    {
     "team_name" : "xyz",
     "type" : "men",
    "Players" : [
           {
          "first" : "John",
          "last" :  "Smith",
          "country_ID" : "123"
        },
        {
          "first" : "Alice",
          "last" :  "Smith",
          "country_ID" : "679"
        }
    ]

    }


**GET** nestedindex/_search

 {
	"query": {

          "bool": {"must": [   {
		"multi_match": {
            "query":"Smith",
            
            "fields":["SINGLE_FIELD"]
			
		}
          }
          ],
          
            "filter": [
    {
      "term": {
        "players.country_ID": {
          "value": "679"
        }
      }
    }
  ]
          
          
          }

	}
}

dadoonet · April 26, 2019, 9:06am

Your example does not work.

GET nestedindex/_search
{
  "query": {
    "match": {
      "SINGLE_FIELD": "Smith"
    }
  }
}

Probably because of a typo with Players vs players.

I changed your "script" by removing all the non needed characters **, removing blank lines, ...
I also added a nested query to make your initial query works.

Not sure what you are looking for then.

DELETE nestedindex
PUT nestedindex
{
  "mappings": {
    "_doc": {
      "properties": {
        "SINGLE_FIELD": {
          "type": "text"
        },
        "team_name": {
          "type": "text",
          "copy_to": [
            "SINGLE_FIELD"
          ]
        },
        "type": {
          "type": "text",
          "copy_to": [
            "SINGLE_FIELD"
          ]
        },
        "players": {
          "type": "nested",
          "properties": {
            "first": {
              "type": "text",
              "copy_to": [
                "SINGLE_FIELD"
              ]
            },
            "last": {
              "type": "text",
              "copy_to": [
                "SINGLE_FIELD"
              ]
            },
            "country_ID": {
              "type": "text",
              "copy_to": [
                "SINGLE_FIELD"
              ]
            }
          }
        }
      }
    }
  }
}
POST nestedindex/_doc/1
{
  "team_name": "xyz",
  "type": "men",
  "players": [
    {
      "first": "John",
      "last": "Smith",
      "country_ID": "123"
    },
    {
      "first": "Alice",
      "last": "Smith",
      "country_ID": "679"
    }
  ]
}
GET nestedindex/_search
{
  "query": {
    "bool": {
      "must": [
        {
          "multi_match": {
            "query": "Smith",
            "fields": [
              "SINGLE_FIELD"
            ]
          }
        }
      ],
      "filter": [
        {
          "nested": {
            "path": "players",
            "query": {
              "term": {
                "players.country_ID": {
                  "value": "679"
                }
              }
            }
          }
        }
      ]
    }
  }
}

system · May 24, 2019, 9:07am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Elasticsearch \| Relationship based Index Elasticsearch	4	572	April 3, 2019
Help me understand Elasticsearch structure Elasticsearch	6	336	March 22, 2021
Some questions about schema/mapping/indexes/filters Elasticsearch	3	397	July 6, 2017
Searching within a single nested object Elasticsearch	4	309	July 6, 2017
Filtering based on data in multiple indexes Elasticsearch	2	344	July 6, 2017

Elasticsearch | Relationships based index

Related topics