# Synonym token filter and edge\_ngram tokenizer conflicts **URL:** \ **Category:** Elasticsearch\ **Created:** [September 6, 2019, 12:54am UTC](https://discuss.elastic.co/t/synonym-token-filter-and-edge-ngram-tokenizer-conflicts/198328 "2019-09-06T00:54:54Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![jjarthur](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jjarthur/32/48545_2.png) [@jjarthur](https://discuss.elastic.co/u/jjarthur)\ **Post date:** [September 6, 2019, 12:54am UTC](https://discuss.elastic.co/t/synonym-token-filter-and-edge-ngram-tokenizer-conflicts/198328/1 "2019-09-06T00:54:54Z")
Hello, I'm facing a small issue with my implementation of the `synonym token filter` combined with an `edge_ngram` tokenizer. For example, I have a synonym mapping for `sa => south_australia` and an entry with a field of `sandcastles are fun`. Now if I query Elasticsearch for `south_australia`, it will also return my `sandcastles are fun` entry in the results. This happens because the `edge_ngram` tokenizer has indexed that field as `{s, sa, san, sand, ... }` and so when `south_australia` is mapped to `sa`, it unfortunately gets picked up. I have added a minified version of my settings file that has the same issue: ``` { "settings": { "analysis": { "analyzer": { "synonym": { "tokenizer": "autocomplete", "filter": ["lowercase", "synonym"] } }, "filter": { "synonym": { "type": "synonym", "synonyms_path": "synonyms" } }, "tokenizer": { "autocomplete": { "type": "edge_ngram", "min_gram": 2, "max_gram": 20, "token_chars": [ "letter", "digit" ] } } } }, "mappings": { "properties": { "title": { "type": "text", "analyzer": "standard", "fields": { "synonyms": { "type": "text", "analyzer": "synonym" } } } } } } ``` Curious if anyone has dealt with this before or has some advice to get around it? ---
**Author:** ![cbuescher](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cbuescher/32/60402_2.png) [@cbuescher](https://discuss.elastic.co/u/cbuescher)\ **Post date:** [September 6, 2019, 3:57pm UTC](https://discuss.elastic.co/t/synonym-token-filter-and-edge-ngram-tokenizer-conflicts/198328/2 "2019-09-06T15:57:22Z")
As you seem to need the edge\_ngram tokenizer for autocomplete functionality only, would it be possible to use two fields instead, one for the autocomplete functionality and one that uses the synonyms that then can be used to increase search recall? Another option might be to apply synonyms only at query time. ---
**Author:** ![jjarthur](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jjarthur/32/48545_2.png) [@jjarthur](https://discuss.elastic.co/u/jjarthur)\ **Post date:** [September 8, 2019, 10:52pm UTC](https://discuss.elastic.co/t/synonym-token-filter-and-edge-ngram-tokenizer-conflicts/198328/3 "2019-09-08T22:52:15Z")
I was under the impression that the way I had set up my synonyms, I had to apply the synonym token filter at both index time _and_ query time. Thank you for your suggestion -- this seemed to have fixed my problem as well as not breaking anything else in the process (I think.. 🙂) . ---
**Author:** ![jjarthur](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jjarthur/32/48545_2.png) [@jjarthur](https://discuss.elastic.co/u/jjarthur)\ **Post date:** [September 8, 2019, 11:45pm UTC](https://discuss.elastic.co/t/synonym-token-filter-and-edge-ngram-tokenizer-conflicts/198328/4 "2019-09-08T23:45:22Z")
After some more testing, I found that this fixed the cases of incomplete token mapping, e.g. a query for `sou` will no longer map to `sa` (via `sout => south_australia => sa`). However, the problem still persists for a full synonym match, e.g. a query for `south_australia` _will_ still map to `sa`. I'm looking at your other suggestion and not sure I fully understand. Do you mean something like: ```auto { "settings": { "analysis": { "analyzer": { "synonym": { "tokenizer": "standard", "filter": ["lowercase", "synonym"] }, "autocomplete": { "tokenizer": "autocomplete", "filter": ["lowercase"] } }, "filter": { "synonym": { "type": "synonym", "synonyms_path": "synonyms" } }, "tokenizer": { "autocomplete": { "type": "edge_ngram", "min_gram": 2, "max_gram": 20, "token_chars": [ "letter", "digit" ] } } } }, "mappings": { "properties": { "title": { "type": "text", "analyzer": "standard", "fields": { "synonyms": { "type": "text", "analyzer": "synonym" }, "autocomplete": { "type": "text", "analyzer": "autocomplete" } } } } } } ``` ---
**Author:** ![cbuescher](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cbuescher/32/60402_2.png) [@cbuescher](https://discuss.elastic.co/u/cbuescher)\ **Post date:** [September 9, 2019, 12:26pm UTC](https://discuss.elastic.co/t/synonym-token-filter-and-edge-ngram-tokenizer-conflicts/198328/5 "2019-09-09T12:26:01Z")
Yes, at a quick glance your last example looks like what I wanted to point out to you: use two independent analyzers for autocomplete and regular search, so that synonyms don't interfere with edge ngrams. I don't know if this completely works in your use case, but it should make things easier to reason about that if mixing autocomplete functionality and search on the same field Also I wonder if you ever tried the dedicated [completion suggester](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-suggesters.html#completion-suggester) for the autocomplete functionality? Maybe you did and discarded it for some reason, just curious and don't want to move this thread towards suggesters but maybe worth another look. ---
**Author:** ![jjarthur](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jjarthur/32/48545_2.png) [@jjarthur](https://discuss.elastic.co/u/jjarthur)\ **Post date:** [September 9, 2019, 10:21pm UTC](https://discuss.elastic.co/t/synonym-token-filter-and-edge-ngram-tokenizer-conflicts/198328/6 "2019-09-09T22:21:00Z")
I ended up implementing both of your suggestions. It is very similar to the above snippet, except I am only using the synonyms analyzer at search time and only using the autocomplete analyzer at index time. In regards to the completion suggester... to be honest I had no idea that existed! I have just gotten everything working as is, but will keep that in mind for future changes. Thanks for taking the time, I appreciate the feedback! ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [October 7, 2019, 10:21pm UTC](https://discuss.elastic.co/t/synonym-token-filter-and-edge-ngram-tokenizer-conflicts/198328/7 "2019-10-07T22:21:00Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.