# Individual Tokens are not searched when a synonym rule is matched **URL:** \ **Category:** Elasticsearch\ **Created:** [April 22, 2021, 7:47am UTC](https://discuss.elastic.co/t/individual-tokens-are-not-searched-when-a-synonym-rule-is-matched/270933 "2021-04-22T07:47:35Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![jacksumit](https://avatars.discourse-cdn.com/v4/letter/j/e495f1/32.png) [@jacksumit](https://discuss.elastic.co/u/jacksumit)\ **Post date:** [April 22, 2021, 7:47am UTC](https://discuss.elastic.co/t/individual-tokens-are-not-searched-when-a-synonym-rule-is-matched/270933/1 "2021-04-22T07:47:36Z")
Elasticsearch 7.10.2 Description: When using synonym\_graph filter, and if a synonym rule is matched, Elastic is not searching for individual tokens. I have defined below synonym rule. ```auto "synonyms": [ "Country Federal Police, CFP" ] ``` When I search for "Country Federal Police", I get documents containing "CFP" and "Country Federal Police" (All 3 words together). It does not match documents containing "Country" , "Federal" and "Police" tokens individually. However if I remove the synonym rule, document containing any 1 of the 3 tokens are also returned. I am using standard tokenizer. Is this expected behaviour? I would expect it to consider individual tokens of original search string. Step to reproduce: Mapping ```auto PUT testsynonymgraph { "settings": { "analysis": { "filter": { "search_synonyms": { "type": "synonym_graph", "synonyms": [ "Country Federal Police, CFP" ] } }, "analyzer": { "default_search": { "filter": ["lowercase", "asciifolding", "search_synonyms", "stop", "kstem"], "type": "custom", "tokenizer": "standard" }, "default": { "filter": ["lowercase", "asciifolding", "stop", "kstem"], "type": "custom", "tokenizer": "standard" } } } }, "mappings": { "properties": { "Name": { "type": "text" } } } } ``` Indexing 4 documents ```auto POST _bulk { "index" : { "_index" : "testsynonymgraph", "_id" : "1" } } { "FaCS Name" : "Country Federal Police" } { "index" : { "_index" : "testsynonymgraph", "_id" : "2" } } { "FaCS Name" : "Country Defence Force" } { "index" : { "_index" : "testsynonymgraph", "_id" : "3" } } { "FaCS Name" : "Country Reserve Police" } { "index" : { "_index" : "testsynonymgraph", "_id" : "4" } } { "FaCS Name" : "CFP" } ``` Search: ```auto POST /testsynonymgraph/_search { "query": { "bool": { "must": [ { "multi_match": { "query": "Country Federal Police", "fuzziness": "Auto" } } ] } }, "size": 10 } ``` Search Result: ```auto "hits" : [ { "_index" : "testsynonymgraph", "_type" : "_doc", "_id" : "1", "_score" : 2.08334, "_source" : { "FaCS Name" : "Country Federal Police" } }, { "_index" : "testsynonymgraph", "_type" : "_doc", "_id" : "4", "_score" : 1.5956267, "_source" : { "FaCS Name" : "CFP" } } ] ``` Problems: Other documents "Country Federal Police" and "Country Reserve Police" should also be returned in result. ---
**Author:** ![Mark\_Harwood](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mark_harwood/32/10538_2.png) [@Mark\_Harwood](https://discuss.elastic.co/u/Mark_Harwood)\ **Post date:** [April 22, 2021, 10:57am UTC](https://discuss.elastic.co/t/individual-tokens-are-not-searched-when-a-synonym-rule-is-matched/270933/2 "2021-04-22T10:57:42Z")
I'd consider that to be desirable behaviour. The synonym rule defines that multiple words collectively mean a single thing e.g. `eye liner` and `eyeliner` both define the thing Robert Smith puts on his face and shouldn't match a `cruise liner`. There are [many of these](https://gist.github.com/markharwood/e521ae9e518c4b275474b201284e552b) compound words and the multi-word version is treated as a single entity. If you want the partial matching behaviour still use a sub-field with a different choice of Analyzer and query both fields (the one with synonyms and the one without). ---
**Author:** ![jacksumit](https://avatars.discourse-cdn.com/v4/letter/j/e495f1/32.png) [@jacksumit](https://discuss.elastic.co/u/jacksumit)\ **Post date:** [April 23, 2021, 12:37am UTC](https://discuss.elastic.co/t/individual-tokens-are-not-searched-when-a-synonym-rule-is-matched/270933/3 "2021-04-23T00:37:11Z")
> [@Mark\_Harwood](#): > > aviour still use a sub-field with a different choice of Analyzer and query both fields (the one with synonyms and the one without). Thanks for your response Mark. I get the point. I added another document as below. ```auto { "index" : { "_index" : "testsynonymgraph", "_id" : "6" } } { "FaCS Name" : "DFP" } ``` On below query, it is applying fuzziness for single word synonyms. ```auto POST /testsynonymgraph/_search { "query": { "bool": { "must": [ { "multi_match": { "query": "Country Federal Police", "fuzziness": "Auto" } } ] } }, "size": 10 } ``` Query Result: DFP is returned in result (I assume, fuzziness is applied on synonym cfp). I thought fuzziness is not applied when synonym rule is matched. It is discussed here as well: [https://github.com/elastic/elasticsearch/issues/25518](https://github.com/elastic/elasticsearch/issues/25518) ```auto "hits" : [ { "_index" : "testsynonymgraph", "_type" : "_doc", "_id" : "4", "_score" : 2.9233453, "_source" : { "FaCS Name" : "CF Police" }, "highlight" : { "FaCS Name" : [ "CF Police" ] } }, { "_index" : "testsynonymgraph", "_type" : "_doc", "_id" : "1", "_score" : 2.395439, "_source" : { "FaCS Name" : "Country Federal Police" }, "highlight" : { "FaCS Name" : [ "Country Federal Police" ], "FaCS Name.keyword" : [ "Country Federal Police" ] } }, { "_index" : "testsynonymgraph", "_type" : "_doc", "_id" : "6", "_score" : 1.3402741, "_source" : { "FaCS Name" : "DFP" }, "highlight" : { "FaCS Name" : [ DFP ] } } ] ``` ---
**Author:** ![Mark\_Harwood](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mark_harwood/32/10538_2.png) [@Mark\_Harwood](https://discuss.elastic.co/u/Mark_Harwood)\ **Post date:** [April 23, 2021, 12:51pm UTC](https://discuss.elastic.co/t/individual-tokens-are-not-searched-when-a-synonym-rule-is-matched/270933/4 "2021-04-23T12:51:24Z")
The docs are not fully clear on this. If you add this doc to your index you should see the discrepancy: ```auto { "index" : { "_index" : "testsynonymgraph", "_id" : "7" } } { "Name" : "County Federal Police" } ``` Note the use of `County` instead of `Country` This doesn't fuzzy-match your query whereas DFP does. That's because multi-word synonyms (Country+Federal+Police) are run as _phrase_ queries whereas single-word synonyms (like CFP) can use fuzzy. ---
**Author:** ![jacksumit](https://avatars.discourse-cdn.com/v4/letter/j/e495f1/32.png) [@jacksumit](https://discuss.elastic.co/u/jacksumit)\ **Post date:** [April 29, 2021, 4:50am UTC](https://discuss.elastic.co/t/individual-tokens-are-not-searched-when-a-synonym-rule-is-matched/270933/5 "2021-04-29T04:50:08Z")
> [@Mark\_Harwood](#): > > zzy-match your query whereas DFP doe Thanks Mark for clarifying this. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [May 27, 2021, 4:50am UTC](https://discuss.elastic.co/t/individual-tokens-are-not-searched-when-a-synonym-rule-is-matched/270933/6 "2021-05-27T04:50:28Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.