# Synonms Analyzer/Filter does not work - Solved!

**URL:** <https://discuss.elastic.co/t/synonms-analyzer-filter-does-not-work-solved/140020>\
**Category:** Elasticsearch\
**Created:** [July 14, 2018, 8:01am UTC](https://discuss.elastic.co/t/synonms-analyzer-filter-does-not-work-solved/140020 "2018-07-14T08:01:39Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![ArunANayagam](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/arunanayagam/32/45171_2.png) [@ArunANayagam](https://discuss.elastic.co/u/ArunANayagam)\
**Post date:** [July 14, 2018, 8:01am UTC](https://discuss.elastic.co/t/synonms-analyzer-filter-does-not-work-solved/140020/1 "2018-07-14T08:01:40Z")

</div>

Hi,

I created a simple synonm analyzer/filter and I was able to apply to the index fine, at least it looks like its applied fine, here's the cluster state extract,

```
     "sports-2018.07.13": {
                "state": "open",
                "settings": {
                    "index": {
                        "creation_date": "1531510636226",
                        "uuid": "teSeXoRQS4WBkM7ylKzgdw",
                        "analysis": {
                            "filter": {
                                "synonym_filter": {
                                    "type": "synonym",
                                    "synonyms": [
                                        "Manchester United, Man U, Man Utd"
                                    ],
                                    "tokenizer": "keyword"
                                }
                            },
                            "analyzer": {
                                "synonym_analyzer": {
                                    "filter": [
                                        "synonym_filter"
                                    ],
                                    "tokenizer": "keyword"
                                }
                            }
                        },
                        "number_of_replicas": "1",
                        "number_of_shards": "5",
                        "version": {
                            "created": "2040099"
                        }
                    }
                },
                "mappings": {...

```

I know for sure there are records because I see them in Kibana.

But when I query like this, I am not getting any results back,

```
{
   "query":{
      "filtered":{
         "query":{
            "multi_match":{
               "query":"Man Utd",
               "fields":[
                  "event.eventName"
               ],
               "analyzer":"synonym_analyzer"
            }
         }
      }
   }
}

```

Any help is appreciated.

Thanks,  
Arun

---

<div class="post-metadata">

**Author:** ![johtani](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/johtani/32/44956_2.png) [@johtani](https://discuss.elastic.co/u/johtani)\
**Post date:** [July 15, 2018, 3:41am UTC](https://discuss.elastic.co/t/synonms-analyzer-filter-does-not-work-solved/140020/2 "2018-07-15T03:41:44Z")

</div>

Hi,

Could you put your mapping of `"event.eventName"`?  
I think the analyzer of `event.eventName` is not keyword analyzer/synonym\_analyzer.

If the analyzer is standard, "Man Utd" is analyzed every single words, "man" and "utd".

---

<div class="post-metadata">

**Author:** ![ArunANayagam](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/arunanayagam/32/45171_2.png) [@ArunANayagam](https://discuss.elastic.co/u/ArunANayagam)\
**Post date:** [July 15, 2018, 10:45am UTC](https://discuss.elastic.co/t/synonms-analyzer-filter-does-not-work-solved/140020/3 "2018-07-15T10:45:56Z")

</div>

Hi Jun,

You are right. I didn't realise you have to set the analyzer of event.eventName to keyword/synonym\_analyzer.  
I have not set any analyzer explicitly,

```
"event": {
    "properties": {
        "eventName": {
            "type": "string"
        }

```

Should I update the mapping with a "keyword" analyzer for that field?

Thanks,  
Arun

---

<div class="post-metadata">

**Author:** ![johtani](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/johtani/32/44956_2.png) [@johtani](https://discuss.elastic.co/u/johtani)\
**Post date:** [July 15, 2018, 11:48am UTC](https://discuss.elastic.co/t/synonms-analyzer-filter-does-not-work-solved/140020/4 "2018-07-15T11:48:10Z")

</div>

Yes, you should use keyword field.  
Unfortunately, you cannot update existing field mapping.  
[https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-put-mapping.html#updating-field-mappings](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-put-mapping.html#updating-field-mappings)

You should reindex your data. using [reindex api](https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-reindex.html#docs-reindex) or index from source.  
Note, you have to create index with fields that you want to use synonym\_analyzer before you index your data.

---

<div class="post-metadata">

**Author:** ![ArunANayagam](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/arunanayagam/32/45171_2.png) [@ArunANayagam](https://discuss.elastic.co/u/ArunANayagam)\
**Post date:** [July 15, 2018, 12:27pm UTC](https://discuss.elastic.co/t/synonms-analyzer-filter-does-not-work-solved/140020/5 "2018-07-15T12:27:01Z")

</div>

Hi @johtani ,

Updating the mapping is fine.

Is changing just the type to be keyword, enable me to use the synonym\_analyzer?

```
"event": {
    "properties": {
        "eventName": {
            "type": "keyword"
        }

```

Please note that the eventName, will be something like this,

> Man Utd vs Arsenal

Can I use type keyword on event.eventName and still be able to use the synonym\_analyzer to find,  
`event.eventName: "Manchester United"` or `event.eventName: "Man U"`

Thanks,  
Arun

---

<div class="post-metadata">

**Author:** ![johtani](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/johtani/32/44956_2.png) [@johtani](https://discuss.elastic.co/u/johtani)\
**Post date:** [July 15, 2018, 1:24pm UTC](https://discuss.elastic.co/t/synonms-analyzer-filter-does-not-work-solved/140020/6 "2018-07-15T13:24:50Z")

</div>

In your use case, you shouldn't use keyword tokenizer any more.  
If use keyword tokenizer, you cannot search "vs" text.  
Thee is a [good documentation](https://www.elastic.co/guide/en/elasticsearch/guide/current/synonyms.html), unfortunately the version is old. But you can understand the concept.  
And also [another good documentation of inverted index](https://www.elastic.co/guide/en/elasticsearch/guide/current/mapping-analysis.html)

---

<div class="post-metadata">

**Author:** ![ArunANayagam](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/arunanayagam/32/45171_2.png) [@ArunANayagam](https://discuss.elastic.co/u/ArunANayagam)\
**Post date:** [July 15, 2018, 2:13pm UTC](https://discuss.elastic.co/t/synonms-analyzer-filter-does-not-work-solved/140020/7 "2018-07-15T14:13:14Z")

</div>

Hi @johtani,

I will definitely go through those links and try understand.  
In the meanwhile, I am hoping you understood my requirement, with that in mind, would you be able to suggest any tokenizer that I can use?

Thanks,  
Arun

---

<div class="post-metadata">

**Author:** ![johtani](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/johtani/32/44956_2.png) [@johtani](https://discuss.elastic.co/u/johtani)\
**Post date:** [July 15, 2018, 2:43pm UTC](https://discuss.elastic.co/t/synonms-analyzer-filter-does-not-work-solved/140020/8 "2018-07-15T14:43:22Z")

</div>

Unfortunately, I don't understand completely.  
I think "standard" or "whitespace" tokenizer is fine.

Also [the standard analyzer](https://www.elastic.co/guide/en/elasticsearch/reference/current/analysis-standard-analyzer.html) would be helpful.

And another consideration point is multi-word synonym or not.  
See [synonym graph token filter.](https://www.elastic.co/guide/en/elasticsearch/reference/current/analysis-synonym-graph-tokenfilter.html)

---

<div class="post-metadata">

**Author:** ![ArunANayagam](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/arunanayagam/32/45171_2.png) [@ArunANayagam](https://discuss.elastic.co/u/ArunANayagam)\
**Post date:** [July 15, 2018, 4:26pm UTC](https://discuss.elastic.co/t/synonms-analyzer-filter-does-not-work-solved/140020/9 "2018-07-15T16:26:05Z")

</div>

Hi @johtani,

Just so I clarify the requirement.

I have a event.eventName field that can have values like this,

> Manchester United vs Chelsea

> Arsenal vs Man Utd

> Tottenham vs Man U

Unfortunately I am not allowed to change the value during index time.  
So I thought I could write a query time synonym analyzer to query all combinations of Manchester United.

Now I am not sure if I require a multi-word synonym.  
The more I go through your links, my requirement looks more and more complex 🙂

Thanks,  
Arun

---

<div class="post-metadata">

**Author:** ![ArunANayagam](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/arunanayagam/32/45171_2.png) [@ArunANayagam](https://discuss.elastic.co/u/ArunANayagam)\
**Post date:** [July 18, 2018, 4:05pm UTC](https://discuss.elastic.co/t/synonms-analyzer-filter-does-not-work-solved/140020/10 "2018-07-18T16:05:19Z")

</div>

@johtani Thank you for pointing me in the right direction.

Multi word synonyms and associated issues with Lucene are well documented here.

> **[Lucene's TokenStreams are actually graphs!](https://blog.mikemccandless.com/2012/04/lucenes-tokenstreams-are-actually.html)**
>
> Lucene's TokenStream class produces the sequence of tokens to be indexed for a document's fields. The API is an iterator: you call increme...

Just for the benefit of all, here are the steps I had to take to define the dictionary and be able query multi word synonyms using a filter type of "synonym\_graph"

1. Close the index, to define the analyzer,

> POST {{server}}/sportsbook-event/\_close

1. Add the analyzer/filter,

> ```
> PUT {{server}}/sportsbook-event/_settings
> {
> "settings":{
> "analysis":{
> "analyzer":{
> "event_name_synonym_analyzer":{
> "tokenizer":"standard",
> "filter":[
> "lowercase",
> "event_name_synonym_filter"
> ]
> }
> },
> "filter":{
> "event_name_synonym_filter":{
> "type":"synonym_graph",
> "synonyms":[
> "manchester united,manchester utd,man u",
> "new york, ny"
> ]
> }
> }
> }
> }
> }
> 
> ```

1. Open the index,

> {{server}}/sportsbook-event/\_open

1. An explain of query vs match\_phrase is quite interesting and tells you what exactly is being searched,

match\_phrase :

> GET sportsbook-event/\_validate/query?explain

> ```
> {
> "query": {
> "match_phrase": {
> "event.eventName": {
> "query": "ny",
> "analyzer": "event_name_synonym_analyzer"
> }
> }
> }
> }
> 
> ```

```
{
  "_shards": {
    "total": 1,
    "successful": 1,
    "failed": 0
  },
  "valid": true,
  "explanations": [
    {
      "index": "sportsbook-event",
      "valid": true,
      "explanation": "spanOr([spanNear([event.eventName:new, event.eventName:york], 0, true), event.eventName:ny])"
    }
  ]
}

```

Query:

```
GET sportsbook-event/_validate/query?explain
{
  "query": {
    "query_string": {
      "default_field": "event.eventName",
      "query": "new york",
      "analyzer": "event_name_synonym_analyzer"
    }
  }
}

{
  "_shards": {
    "total": 1,
    "successful": 1,
    "failed": 0
  },
  "valid": true,
  "explanations": [
    {
      "index": "sportsbook-event",
      "valid": true,
      "explanation": """(event.eventName:ny event.eventName:"new york")"""
    }
  ]
}

```

Search results seems to be quite accurate.

Thanks,  
Arun

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [August 15, 2018, 4:09pm UTC](https://discuss.elastic.co/t/synonms-analyzer-filter-does-not-work-solved/140020/11 "2018-08-15T16:09:26Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.