Elasticsearch | Relationships based index


(Nikesh) #1

As asked in this link
I need to provide relationship based search.
Let's consider the same scenario.
Team and Players would contain similar fields.
A single team can have multiple players.
Having a Single Index for this would be something of this structure :

 {
 "Team_name" : "xyz",
 "type" : "men",
"Players" : [
       {
      "first" : "John",
      "last" :  "Smith",
      "country_ID" : "123"
    },
    {
      "first" : "Alice",
      "last" :  "Smith",
      "country_ID" : "679"
    }
]

}

I have few doubts over this architecture :

  1. Any update to a particular part of "Players" array. Let's say " first" : "John" is changed to "first":"Mark"
    This updating procedure requires reindexing the entire nested document.
    Is there a procedure to Update just the array element to which it belongs to?

  2. I would be using copy_to field and search would be based on accessibility and this is done through "country_ID". If a user has access to "Country_ID" :"679" and not to "123".
    If a user searches for a string "Smith", will "Smith" with "country_ID" :"123"
    also be searched and highlighted? If so, how do we prevent it from happening?

  3. Suppose one player has to be deleted, let's say Alice Smith , Is there a procedure to just delete this player or does it require reindexing of this document without details of Alice Smith?


(Nikesh) #2

@dadoonet Apologies for creating new thread as link has been closed.
Please provide any suggestions


(David Pilato) #3
  1. No
  2. Not sure I understood but it's easy with Elasticsearch and Kibana dev console to test it.
  3. You need to reindex the whole document.

(Nikesh) #4
  1. My mapping is defined such that there exists a copy_to field enabled. This means Nested Object "Players" and all the other fields part of this JSON are also part of copy_to field.
    Users might have access to search on only few players. For example, I have access to search only on "John Smith" and not "Alice Smith". This accessibility is based on "country_ID", which is part of my query through filter. I am using the "copy_to" field to search.
    Incase I search on "Smith", I have to make sure it searches only "Smith" with "country_ID" :"123" and not "country_ID" : "679 and only the former should be part of the result.

(David Pilato) #5

Could you provide a full recreation script as described in About the Elasticsearch category. It will help to better understand what you are doing. Please, try to keep the example as simple as possible.

A full reproduction script will help readers to understand, reproduce and if needed fix your problem. It will also most likely help to get a faster answer.


(Nikesh) #6

Thanks for the response.
Sure!

This is my mapping :

{
  "mappings": {
    "_doc": {
      "properties": {
      	"SINGLE_FIELD": {
                        "type": "text"
                        
                    },
      	"team_name" : {
      		"type": "text",
      		"copy_to": ["SINGLE_FIELD"]
      		
      	},
      "type" : {
      "type": "text",
     "copy_to": ["SINGLE_FIELD"]
      },
        "players": {
          "type": "nested",
          "properties": {
             "first" : {
        			"type" : "text",
        			"copy_to": ["SINGLE_FIELD"]
             },
             "last":{
             	"type" : "text",
             	"copy_to": ["SINGLE_FIELD"]
             },
             "country_ID" :{
             	"type" : "text",
             	"copy_to": ["SINGLE_FIELD"]
             }
          }
        }
        
      }
    }
  }
}

And I have indexed one document :

{
 "team_name" : "xyz",
 "type" : "men",
"Players" : [
       {
      "first" : "John",
      "last" :  "Smith",
      "country_ID" : "123"
    },
    {
      "first" : "Alice",
      "last" :  "Smith",
      "country_ID" : "679"
    }
]

}

My query is :

{
	"query": {

         
		"multi_match": {
            "query":"Smith",
            
            "fields":["SINGLE_FIELD"]
			
		}

	}
} 

Now if I as a user has access to only player with country_ID : 679 and not to 123.
How do I restrict my search and highlight on country_ID : 123?


(David Pilato) #7

Use a bool query.
Add the multi_match within the must clause and add a term query on country_ID field within the filter clause.


(Nikesh) #8

thanks for your response.

You meant query structure of the below sort?

{
  "query": {
    "bool": {
      "must": [
        {
          "multi_match": {
            "query": "Smith",
            "fields": [
              "SINGLE_FIELD"
            ]
          }
        }
      ],
      "filter": [
        {
          "term": {
            "country_ID": {
              "value": "679"
            }
          }
        }
      ]
    }
  }
}

(David Pilato) #9

Yes.


(Nikesh) #10

Thanks for the response.
But this is not providing me any output.
Although the below query without any filter fetches me results -

{
  "query": {
    "bool": {
      "must": [
        {
          "multi_match": {
            "query": "Smith",
            "fields": [
              "SINGLE_FIELD"
            ]
          }
        }
      ]
    }
  }
} 

Is something wrong with my filter clause?
this is my filter clause -

  "filter": [
    {
      "term": {
        "country_ID": {
          "value": "679"
        }
      }
    }
  ]

(David Pilato) #11

Could you provide a full recreation script as described in About the Elasticsearch category. It will help to better understand what you are doing. Please, try to keep the example as simple as possible.

A full reproduction script will help readers to understand, reproduce and if needed fix your problem. It will also most likely help to get a faster answer.


(Nikesh) #12

Yes. The below is my query. This does not output any result.

   {
	"query": {

          "bool": {"must": [   {
		"multi_match": {
            "query":"Smith",
            
            "fields":["SINGLE_FIELD"]
			
		}
          }
          ],
          
            "filter": [
    {
      "term": {
        "players.country_ID": {
          "value": "679"
        }
      }
    }
  ]
          
          
          }

	}
}

(David Pilato) #13

There's no way to reproduce the problem without a full script. Unless someone wants to assemble it from the snippets you provided.
I'd recommend providing it yourself as described per the link I shared.


(Nikesh) #14

This is my mapping :

{
  "mappings": {
    "_doc": {
      "properties": {
      	"SINGLE_FIELD": {
                        "type": "text"
                        
                    },
      	"team_name" : {
      		"type": "text",
      		"copy_to": ["SINGLE_FIELD"]
      		
      	},
      "type" : {
      "type": "text",
     "copy_to": ["SINGLE_FIELD"]
      },
        "players": {
          "type": "nested",
          "properties": {
             "first" : {
        			"type" : "text",
        			"copy_to": ["SINGLE_FIELD"]
             },
             "last":{
             	"type" : "text",
             	"copy_to": ["SINGLE_FIELD"]
             },
             "country_ID" :{
             	"type" : "text",
             	"copy_to": ["SINGLE_FIELD"]
             }
          }
        }
        
      }
    }
  }
}

And I have indexed one document :

{
 "team_name" : "xyz",
 "type" : "men",
"Players" : [
       {
      "first" : "John",
      "last" :  "Smith",
      "country_ID" : "123"
    },
    {
      "first" : "Alice",
      "last" :  "Smith",
      "country_ID" : "679"
    }
]

}

The below is my query. This does not output any result.

   {
	"query": {

          "bool": {"must": [   {
		"multi_match": {
            "query":"Smith",
            
            "fields":["SINGLE_FIELD"]
			
		}
          }
          ],
          
            "filter": [
    {
      "term": {
        "players.country_ID": {
          "value": "679"
        }
      }
    }
  ]
          
          
          }

	}
}

Now if I as a user has access to only player with country_ID : 679 and not to 123 .
How do I restrict my search and highlight on country_ID : 123 ?


(Nikesh) #15

@dadoonet Please provide your suggestion


(David Pilato) #16

@Nikesh let me paste the documentation I linked to in case you did not read it yet.

A typical script like this one can be copied and pasted in Kibana Dev Console by any reader. It will definitely help to play with your example and provide a fix for your script.
If you don't provide it, there is a chance that nobody can help.

Emphasis is mine.

Here is a typical script that anyone can use for example:

DELETE index
PUT index/_doc/1
{
  "foo": "bar"
}
GET index/_search
{
  "query": {
    "match": {
      "foo": "bar"
    }
  }
}

So if you need more help, please provide this.


(Nikesh) #17

Okay!

PUT nestedindex

{
  "mappings": {
    "_doc": {
      "properties": {
      	"SINGLE_FIELD": {
                        "type": "text"
                        
                    },
      	"team_name" : {
      		"type": "text",
      		"copy_to": ["SINGLE_FIELD"]
      		
      	},
      "type" : {
      "type": "text",
     "copy_to": ["SINGLE_FIELD"]
      },
        "players": {
          "type": "nested",
          "properties": {
             "first" : {
        			"type" : "text",
        			"copy_to": ["SINGLE_FIELD"]
             },
             "last":{
             	"type" : "text",
             	"copy_to": ["SINGLE_FIELD"]
             },
             "country_ID" :{
             	"type" : "text",
             	"copy_to": ["SINGLE_FIELD"]
             }
          }
        }
        
      }
    }
  }
}

POST nestedindex/_doc/1

{
 "team_name" : "xyz",
 "type" : "men",
"Players" : [
       {
      "first" : "John",
      "last" :  "Smith",
      "country_ID" : "123"
    },
    {
      "first" : "Alice",
      "last" :  "Smith",
      "country_ID" : "679"
    }
]

}

GET nestedindex/_search

 {
	"query": {

          "bool": {"must": [   {
		"multi_match": {
            "query":"Smith",
            
            "fields":["SINGLE_FIELD"]
			
		}
          }
          ],
          
            "filter": [
    {
      "term": {
        "players.country_ID": {
          "value": "679"
        }
      }
    }
  ]
          
          
          }

	}
}