# Unable to bypass restriction with synonym token filter

**URL:** <https://discuss.elastic.co/t/unable-to-bypass-restriction-with-synonym-token-filter/195334>\
**Category:** Elasticsearch\
**Created:** [August 15, 2019, 11:44am UTC](https://discuss.elastic.co/t/unable-to-bypass-restriction-with-synonym-token-filter/195334 "2019-08-15T11:44:11Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![radoslav.sholev](https://avatars.discourse-cdn.com/v4/letter/r/58f4c7/32.png) [@radoslav.sholev](https://discuss.elastic.co/u/radoslav.sholev)\
**Post date:** [August 15, 2019, 11:44am UTC](https://discuss.elastic.co/t/unable-to-bypass-restriction-with-synonym-token-filter/195334/1 "2019-08-15T11:44:11Z")

</div>

Greetings,

I am trying to migrate from ES 5.5 to 7.3, but I got stuck at an issue with the synonym filter. The reference states "_This filter tokenize synonyms with whatever tokenizer and token filters appear before it in the chain._" . From it I understand that when I define a custom analyzer, the synonyms will get tokenized and filtered by previous token filters, but I don't want that. I need my synonims untouched. Here is a little example to show what I am doing:

```
PUT index_1
{
  "mappings": {
	"properties": {
	  "text": {
		"analyzer": "custom_analyzer_all",
		"norms": false,
		"type": "text"
	  }
	}
  },
  "settings": {
	"index.number_of_replicas": 1,
	"analysis": {
	  "analyzer": {
		"custom_analyzer_all": {
		  "char_filter": [
		  ],
		  "filter": [
			"lowercase",
			"custom_lemma",
			"custom_wordpack"
		  ],
		  "tokenizer": "standard",
		  "type": "custom"
		}
	  },
	  "char_filter": {
	  },
	  "filter": {
		"custom_wordpack": {
		  "type": "synonym",
		  "tokenizer": "whitespace",
		  "synonyms": [
			"gym => gym, _amenities"
		  ]
		},
		"custom_lemma": {
		  "type": "lemmagen",
		  "lexicon": "en"
		}
	  }
	},
	"index.number_of_shards": 1
  }
}

```

I am creating an index with a custom analyzer that uses lowercase, synonym and lemmagen filter. Lemmagen is a plugin([Link to repo](https://github.com/vhyza/elasticsearch-analysis-lemmagen) ).  
Insert a sample document.

```
POST index_1/_doc/1
{
  "text":"Great to see Ivan can get things right in a gym - clean and everything works."
}

```

Now when I check the terms of the text field with \_termvectors API:

```
POST index_1/_termvectors/1
{
  "fields": ["text"]
}

```

I get this for the added synonym:

```
{
	"_amenity" : {
	  "term_freq" : 1,
	  "tokens" : [
		{
		  "position" : 10,
		  "start_offset" : 44,
		  "end_offset" : 47
		}
	  ]
	}
}

```

"\_amenity" is the lemmatized version of the synonym "\_amenities". Even if I swap the places of the custom\_lemma and custom\_wordpack filters, it will still have the same effect(the synonym is inserted in its original form, but the custom\_lemma filter again lematizes it). I need somehow to prevent the synonyms from getting analyzed by the filters or tokenizer. It used to be possible to specify a tokenizer inside the synonym token filter, but that no longer works. Could someone please suggest anything on how to have the text field analyzed as it is, but the synonyms are left untouched.

---

<div class="post-metadata">

**Author:** ![William\_Brafford](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/william_brafford/32/51559_2.png) [@William\_Brafford](https://discuss.elastic.co/u/William_Brafford)\
**Post date:** [August 15, 2019, 10:18pm UTC](https://discuss.elastic.co/t/unable-to-bypass-restriction-with-synonym-token-filter/195334/2 "2019-08-15T22:18:32Z")

</div>

Welcome to the Elasticsearch forums! 👋

This is an interesting case. It looks like the behavior changed with 6.0.0. (See under "Analysis" in the [release notes' New Features](https://www.elastic.co/guide/en/elasticsearch/reference/6.0/release-notes-6.0.0.html#feature-6.0.0).) You're right about the way that synonyms go through the entire token filter chain regardless of where the synonym filter is placed. For anyone following along, you can reproduce this behavior without installing any plugins by using a `stemmer` token filter:

```auto
PUT index_2
{
  "mappings": {
    "properties": {
      "text": {
        "analyzer": "stems_then_synonyms",
        "norms": false,
        "type": "text"
      }
    }
  },
  "settings": {
    "index.number_of_replicas": 1,
    "analysis": {
      "analyzer": {
        "stems_then_synonyms": {
          "char_filter": [],
          "filter": [
            "lowercase",
            "my_stemmer",
            "my_synonyms"
          ],
          "tokenizer": "standard",
          "type": "custom"
        },
        "synonyms_then_stems": {
          "char_filter": [],
          "filter": [
            "lowercase",
            "my_synonyms",
            "my_stemmer"
          ],
          "tokenizer": "standard",
          "type": "custom"
        }
      },
      "char_filter": {},
      "filter": {
        "my_synonyms": {
          "type": "synonym",
          "tokenizer": "whitespace",
          "synonyms": [
            "gym => gym, _amenities"
          ]
        },
        "my_stemmer": {
          "type": "stemmer",
          "name": "english"
        }
      }
    },
    "index.number_of_shards": 1
  }
}

POST index_2/_analyze
{
  "analyzer": "stems_then_synonyms",
  "text": "gym"
}

POST index_2/_analyze
{
  "analyzer": "synonyms_then_stems",
  "text": "gym"
}

```

Both of those calls to `_analyze` result in the same thing:

```auto
{
  "tokens" : [
    {
      "token" : "gym",
      "start_offset" : 0,
      "end_offset" : 3,
      "type" : "SYNONYM",
      "position" : 0
    },
    {
      "token" : "_amen",
      "start_offset" : 0,
      "end_offset" : 3,
      "type" : "SYNONYM",
      "position" : 0
    }
  ]
}

```

Can you tell us a little bit more about your use case? Are you using the synonym token filter as a way of adding document tags? I wonder if you could use a [multi field](https://www.elastic.co/guide/en/elasticsearch/reference/7.3/multi-fields.html) to capture both kinds of analysis for every document:

```auto
PUT index_3
{
  "mappings": {
    "properties": {
      "text": {
        "analyzer": "custom_analyzer_lemma",
        "norms": false,
        "type": "text",
        "fields": {
          "synonymized": {
            "analyzer": "custom_analyzer_wordpack",
            "type": "text"
          }
        }
      }
    }
  },
  "settings": {
    "index.number_of_replicas": 1,
    "analysis": {
      "analyzer": {
        "custom_analyzer_lemma": {
          "char_filter": [],
          "filter": [
            "lowercase",
            "custom_lemma"
          ],
          "tokenizer": "standard",
          "type": "custom"
        },
        "custom_analyzer_wordpack": {
          "char_filter": [],
          "filter": [
            "lowercase",
            "custom_wordpack"
          ],
          "tokenizer": "standard",
          "type": "custom"
        }
      },
      "char_filter": {},
      "filter": {
        "custom_wordpack": {
          "type": "synonym",
          "tokenizer": "whitespace",
          "synonyms": [
            "gym => gym, _amenities"
          ]
        },
        "custom_lemma": {
          "type": "lemmagen",
          "lexicon": "en"
        }
      }
    },
    "index.number_of_shards": 1
  }
}

PUT index_3/_doc/1
{
  "text": "cars park at the gym"
}

POST index_3/_termvectors/1
{
  "fields": ["text", "text.synonymized"]
}

```

-William

---

<div class="post-metadata">

**Author:** ![radoslav.sholev](https://avatars.discourse-cdn.com/v4/letter/r/58f4c7/32.png) [@radoslav.sholev](https://discuss.elastic.co/u/radoslav.sholev)\
**Post date:** [August 16, 2019, 7:44am UTC](https://discuss.elastic.co/t/unable-to-bypass-restriction-with-synonym-token-filter/195334/3 "2019-08-16T07:44:21Z")

</div>

Hi @William_Brafford  
Thank you for taking the time to look at my issue. Yes, you are correct. I am using the synonym filter to tag texts and then perform aggregations on that tag. I have tried the multi field approach, but it doesn't work for me. I need to stem/lemmatize the field before matching it with the synonym. In your example with multi field, "synonymized" would hold the original text's terms lowercased only. Let's say the example text doesn't have the word "gym", but its plural form "gyms" instead. In that case the synonym filter won't capture "gyms", but if it is lemmatized then "gyms" would become "gym" and the filter will add the tag to the terms.  
There are other workarounds like putting all the possible forms of the given word in the synonym filter, but I'd rather not do that. Another way is just to use the lemmatized form of the tag instead of whatever I want, but I am still considering this.

---

<div class="post-metadata">

**Author:** ![William\_Brafford](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/william_brafford/32/51559_2.png) [@William\_Brafford](https://discuss.elastic.co/u/William_Brafford)\
**Post date:** [August 16, 2019, 3:45pm UTC](https://discuss.elastic.co/t/unable-to-bypass-restriction-with-synonym-token-filter/195334/4 "2019-08-16T15:45:55Z")

</div>

@radoslav.sholev,

Thanks for explaining the use case. I think I understand the issue better now.

I'm looking at some of the discussions around when this change happened (before my time as an employee), and it seems like we intended the `synonym_graph` search-time filter ([docs here](https://www.elastic.co/guide/en/elasticsearch/reference/7.3/analysis-synonym-graph-tokenfilter.html)) to cover some of the cases that the old `synonym` filter used to cover before 6.0. See, for example, the discussion here:

> [@Confusion in using synonym token filter in elasticsearch 6.x](https://discuss.elastic.co/t/confusion-in-using-synonym-token-filter-in-elasticsearch-6-x/128454/):
>
> Synonym token in elasticsearch v2.4 supports tokenizer parameter. Hence token filter could have its own tokenizer (here "keyword") different from that is being used in custom analyser (here "whitespace") as in below setting: { "settings": { "index": { "analysis": { "filter": { "my\_synonym\_filter": { "type": "synonym", "synonyms\_path": "synonym.txt", "tokenizer": "keyword" } }, "analyzer": { "my…

I tried it out in Kibana. It looks like what I'm doing here doesn't store the tag on the document, but does let you use the tag in search:

```auto
PUT index_4
{
  "mappings": {
    "properties": {
      "text": {
        "analyzer": "custom_index_analyzer",
        "norms": false,
        "type": "text"
      }
    }
  },
  "settings": {
    "index.number_of_replicas": 1,
    "analysis": {
      "analyzer": {
        "custom_index_analyzer": {
          "char_filter": [],
          "filter": [
            "lowercase",
            "custom_lemma"
          ],
          "tokenizer": "standard",
          "type": "custom"
        },
        "custom_search_analyzer": {
          "char_filter": [],
          "filter": [
            "lowercase",
            "custom_wordpack"
          ],
          "tokenizer": "standard",
          "type": "custom"
        }
      },
      "char_filter": {},
      "filter": {
        "custom_wordpack": {
          "type": "synonym_graph",
          "synonyms": [
            "gym, _amenities"
          ]
        },
        "custom_lemma": {
          "type": "lemmagen",
          "lexicon": "en"
        }
      }
    },
    "index.number_of_shards": 1
  }
}

```

At index time, we use the LemmaGen analyzer on the text field. The tag isn't added.

However, when a search for `_amenities` using the search-time analyzer will search for the `gym` token.

```auto
POST index_4/_analyze
{
  "analyzer": "custom_search_analyzer",
  "text": "_amenities"
}

```

Result:

```auto
{
  "tokens" : [
    {
      "token" : "gym",
      "start_offset" : 0,
      "end_offset" : 10,
      "type" : "SYNONYM",
      "position" : 0
    },
    {
      "token" : "_amenities",
      "start_offset" : 0,
      "end_offset" : 10,
      "type" : "<ALPHANUM>",
      "position" : 0
    }
  ]
}

```

Now if we index one document with "gym" and another with "gyms":

```auto
PUT index_4/_doc/1
{
  "text": "nice chairs at the gym"
}

PUT index_4/_doc/2
{
  "text": "there are plenty of gyms in the city"
}

```

Our query will find and highlight both terms:

```auto
GET index_4/_search
{
  "query": {
    "match": {
      "text": {
        "query": "_amenities",
        "analyzer": "custom_search_analyzer"
      }
    }
  },
  "highlight": {
    "fields": {
      "text": {}
    }
  }
}

```

Result:

```auto
{
  [...],
  "hits" : {
    "total" : {
      "value" : 2,
      "relation" : "eq"
    },
    "max_score" : 0.27884474,
    "hits" : [
      {
        "_index" : "index_4",
        "_type" : "_doc",
        "_id" : "1",
        "_score" : 0.27884474,
        "_source" : {
          "text" : "nice chairs at the gym"
        },
        "highlight" : {
          "text" : [
            "nice chairs at the <em>gym</em>"
          ]
        }
      },
      {
        "_index" : "index_4",
        "_type" : "_doc",
        "_id" : "2",
        "_score" : 0.27884474,
        "_source" : {
          "text" : "there are plenty of gyms in the city"
        },
        "highlight" : {
          "text" : [
            "there are plenty of <em>gyms</em> in the city"
          ]
        }
      }
    ]
  }
}

```

And here's a stab at an aggregation that counts documents matching tags:

```auto
GET index_4/_search
{
  "query": {
    "match_all": {}
  },
  "size": 0,
  "aggs": {
    "tag_agg": {
      "filters": {
        "filters": {
          "amenities": {
            "match": {
              "text": {
                "query": "_amenities",
                "analyzer": "custom_search_analyzer"
              }
            }
          },
          "no_amenities": {
            "bool": {
              "must_not": {
                "match": {
                  "text": {
                    "query": "_amenities",
                    "analyzer": "custom_search_analyzer"
                  }
                }
              }
            }
          }
        }
      }
    }
  }
}

```

Result:

```auto
{
  [...],
  "aggregations" : {
    "tag_agg" : {
      "buckets" : {
        "amenities" : {
          "doc_count" : 2
        },
        "no_amenities" : {
          "doc_count" : 0
        }
      }
    }
  }
}

```

I think this approach gets closer to your use-case than the multi-field suggestion does, though I am not sure if it fully solves the problem. Please let me know what you think.

-William

---

<div class="post-metadata">

**Author:** ![radoslav.sholev](https://avatars.discourse-cdn.com/v4/letter/r/58f4c7/32.png) [@radoslav.sholev](https://discuss.elastic.co/u/radoslav.sholev)\
**Post date:** [September 2, 2019, 12:52pm UTC](https://discuss.elastic.co/t/unable-to-bypass-restriction-with-synonym-token-filter/195334/5 "2019-09-02T12:52:57Z")

</div>

Hi @William_Brafford

Sorry for the late response.  
The solution looks great, I haven't considered using a seperate analyzer for searches before and it opens an entirely new perspective for search queries. I have been playing with this to see how I can fit it in my project. Sadly I can't get around the missing term(\_amenities) in the \_termvectors. I need it, since I am using a lot of [span](https://www.elastic.co/guide/en/elasticsearch/reference/7.3/span-queries.html) queries for additional analysis of the texts. I have tried converting the spans into [intervals](https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-intervals-query.html) query, but again hit the same bump, since "match" query requires terms in order to calculate max\_gaps.

---

<div class="post-metadata">

**Author:** ![radoslav.sholev](https://avatars.discourse-cdn.com/v4/letter/r/58f4c7/32.png) [@radoslav.sholev](https://discuss.elastic.co/u/radoslav.sholev)\
**Post date:** [September 5, 2019, 10:18am UTC](https://discuss.elastic.co/t/unable-to-bypass-restriction-with-synonym-token-filter/195334/6 "2019-09-05T10:18:02Z")

</div>

I ended up replacing the the tag with an unique identifier. This way nor the tokenizer, nor the lemmagen filter change the term. In the UI the user can still write his own tag name, but in the backend it is replaced with the ID in order to perform searches and adding new synonym groups to the index. It's a cheap and easy workaround.

Thank you for the support @William_Brafford!

---

<div class="post-metadata">

**Author:** ![William\_Brafford](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/william_brafford/32/51559_2.png) [@William\_Brafford](https://discuss.elastic.co/u/William_Brafford)\
**Post date:** [September 5, 2019, 8:12pm UTC](https://discuss.elastic.co/t/unable-to-bypass-restriction-with-synonym-token-filter/195334/7 "2019-09-05T20:12:00Z")

</div>

@radoslav.sholev ,

I'm glad this worked for you! I learned a lot while thinking through your use case. Thanks for the detailed write-ups.

-William

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 3, 2019, 8:12pm UTC](https://discuss.elastic.co/t/unable-to-bypass-restriction-with-synonym-token-filter/195334/8 "2019-10-03T20:12:01Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.