# Tokenizer works after Synonym (graph) token filter too?

**URL:** <https://discuss.elastic.co/t/tokenizer-works-after-synonym-graph-token-filter-too/280297>\
**Category:** Elasticsearch\
**Created:** [August 3, 2021, 9:53am UTC](https://discuss.elastic.co/t/tokenizer-works-after-synonym-graph-token-filter-too/280297 "2021-08-03T09:53:31Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![taichi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/taichi/32/46431_2.png) [@taichi](https://discuss.elastic.co/u/taichi)\
**Post date:** [August 3, 2021, 9:53am UTC](https://discuss.elastic.co/t/tokenizer-works-after-synonym-graph-token-filter-too/280297/1 "2021-08-03T09:53:31Z")

</div>

I'm using Elasticsearch 7.7 on AWS Elasticsearch Service.

I thought token filters works after tokenizers and tokenizers do nothing after token filters.  
(like `Tokenizers = > Token filters => END`)  
But it seems tokenizers work again after Synonym graph token filters handle synonyms.  
I wonder if it workds like `Tokenizers = > Token filters => Tokenizers? => END`.

I tested like this:

```auto
GET /index/_analyze
{
  "tokenizer": "standard",
  "filter": [{"type": "synonym_graph", "synonyms":["brown fox => brown fox,black cat"]}], 
  "text": "brown fox"
}

```

and got `brown, black, fox, cat`.

```auto
{
    "tokens": [
        {
            "token": "brown",
            "start_offset": 0,
            "end_offset": 9,
            "type": "SYNONYM",
            "position": 0
        },
        {
            "token": "black",
            "start_offset": 0,
            "end_offset": 9,
            "type": "SYNONYM",
            "position": 0,
            "positionLength": 2
        },
        {
            "token": "fox",
            "start_offset": 0,
            "end_offset": 9,
            "type": "SYNONYM",
            "position": 1,
            "positionLength": 2
        },
        {
            "token": "cat",
            "start_offset": 0,
            "end_offset": 9,
            "type": "SYNONYM",
            "position": 2
        }
    ]
}

```

I expected I get only `brown, fox` because `brown fox` in the synonyms does not exists in the output of standard tokenizer.

Another example:

```auto
{
  "tokenizer": "standard",
  "filter": [{"type": "synonym_graph", "synonyms":["fox => fox,black cat"]}], 
  "text": "fox"
}

```

returned `fox, black, cat` but I expected `fox, black cat` because I thought tokenizer does not work after token filter to tokenize `black cat`.

Could you teach me the exact order in which tokenizers and token filters work?

---

<div class="post-metadata">

**Author:** ![vincenbr](https://avatars.discourse-cdn.com/v4/letter/v/8edcca/32.png) [@vincenbr](https://discuss.elastic.co/u/vincenbr)\
**Post date:** [August 3, 2021, 10:06pm UTC](https://discuss.elastic.co/t/tokenizer-works-after-synonym-graph-token-filter-too/280297/2 "2021-08-03T22:06:19Z")

</div>

Hi taichi,  
a text analyzer operates in sequence. it has (in order):

- 0 or more character filters
- exactly one tokenizer
- 0 or more token filters, applied in order  
in your case, the synonym graph token filter is the last operation.  
I suggest you to read about token graph to better understand how token filters work: [Token graphs | Elasticsearch Guide [7.14] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/token-graphs.html)

Furthermore, the synonyms in `synonym_graph` are tokenized and analyzed with the chain which precedes this filter. That explains why "brown fox" synonym is applied. see:  
[[Synonym graph token filter | Elasticsearch Guide [7.14] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/analysis-synonym-graph-tokenfilter.html#_parsing_synonym_files_2)]

---

<div class="post-metadata">

**Author:** ![taichi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/taichi/32/46431_2.png) [@taichi](https://discuss.elastic.co/u/taichi)\
**Post date:** [August 8, 2021, 4:21pm UTC](https://discuss.elastic.co/t/tokenizer-works-after-synonym-graph-token-filter-too/280297/3 "2021-08-08T16:21:36Z")

</div>

Thank you!

> Furthermore, the synonyms in `synonym_graph` are tokenized and analyzed with the chain which precedes this filter

I almost understood that but a little confused. The docs say

> Elasticsearch will use the **token filters** preceding the synonym filter in a tokenizer chain to parse the entries in a synonym file

It just says it applies preceding token filters. Does it apply preceding **tokenizers** to the synonym words too?  
In other words, I wonder if the `"standard"` tokenizer is applied to the synonym words (like `brown fox` and `black cat`).

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [September 5, 2021, 4:21pm UTC](https://discuss.elastic.co/t/tokenizer-works-after-synonym-graph-token-filter-too/280297/4 "2021-09-05T16:21:38Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.