# Setting min=max in ngram tokenizers

**URL:** <https://discuss.elastic.co/t/setting-min-max-in-ngram-tokenizers/344218>\
**Category:** Elasticsearch\
**Created:** [October 2, 2023, 11:10am UTC](https://discuss.elastic.co/t/setting-min-max-in-ngram-tokenizers/344218 "2023-10-02T11:10:05Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![cawoodm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cawoodm/32/14083_2.png) [@cawoodm](https://discuss.elastic.co/u/cawoodm)\
**Post date:** [October 2, 2023, 11:10am UTC](https://discuss.elastic.co/t/setting-min-max-in-ngram-tokenizers/344218/1 "2023-10-02T11:10:05Z")

</div>

The [docs](https://www.elastic.co/guide/en/elasticsearch/reference/current/analysis-ngram-tokenizer.html) suggest setting min=max on ngrams - but this results in a very poor search experience. Assuming 3,3 and searching for "rough" you will never find "trough". Only if you search "tro", "rou" or any other 3 character string will you find something.

> It usually makes sense to set `min_gram` and `max_gram` to the same value. The smaller the length, the more documents will match but the lower the quality of the matches. The longer the length, the more specific the matches. A tri-gram (length `3` ) is a good place to start.

IMHO these values should not be equal but defined as follows:

- min: The shortest string you want to deliver results (typically 3, sometimes 2 depending on the use case)
- max: The longest string you want to deliver results (typically 10, maybe as high as 20 depending on how your users search

I feel the tip does not explain it's reasoning and leads users to frustration since 3,3 is basically useless.

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [October 2, 2023, 11:14am UTC](https://discuss.elastic.co/t/setting-min-max-in-ngram-tokenizers/344218/2 "2023-10-02T11:14:36Z")

</div>

Run a simple test with the example you used and you will see that that is not how ngrams work. The ngram tokenisation is performed both at index and query time (unless you specify a different analyser to be used at search time), so the example you provided will match as several trigrams will be in common.

---

<div class="post-metadata">

**Author:** ![cawoodm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cawoodm/32/14083_2.png) [@cawoodm](https://discuss.elastic.co/u/cawoodm)\
**Post date:** [October 2, 2023, 12:56pm UTC](https://discuss.elastic.co/t/setting-min-max-in-ngram-tokenizers/344218/3 "2023-10-02T12:56:48Z")

</div>

I have of course tested this. (3,3) returns zero results for rough whilst (2,20) does.

Using /\_analyze it's easy to see why (3,3) isn't working:

```auto
POST /_analyze
{
  "tokenizer": {
    "type": "ngram",
    "min_gram": 3,
    "max_gram": 3,
    "token_chars": [
      "letter",
      "digit"
    ]
  },
  "filter": [
    "lowercase"
  ],
  "text": "Trough"
}

And (2,20) is:

tro
rou
oug
ugh

```

```auto
POST /_analyze
{
  "tokenizer": {
    "type": "ngram",
    "min_gram": 2,
    "max_gram": 20,
    "token_chars": [
      "letter",
      "digit"
    ]
  },
  "filter": [
    "lowercase"
  ],
  "text": "Trough"
}

tr
tro
trou
troug
trough
ro
rou
roug
rough
ou
oug
ough
ug
ugh
gh

```

Only with 2,20 do we see "rough" in the inverted index.

However I do understand your point and would be interested how to get it to work. We tested with `POST /_search?q=fulltext.en:rough` and get zero results. Somehow the tokenizer is not being applied at search time.

But to be honest we don't want user's search of "rough" to provide the same results as "rou" so I'm not sure this is a route we want to take.

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [October 2, 2023, 1:24pm UTC](https://discuss.elastic.co/t/setting-min-max-in-ngram-tokenizers/344218/4 "2023-10-02T13:24:23Z")

</div>

With (3,3) ngram both `rough` and `trough` will generate the tokens `rou`, `oug` and `ugh` both at index and query time, which means they will match.

> [@cawoodm](#):
>
> However I do understand your point and would be interested how to get it to work. We tested with `POST /_search?q=fulltext.en:rough` and get zero results. Somehow the tokenizer is not being applied at search time.

What is the mapping for that field?

> [@cawoodm](#):
>
> But to be honest we don't want user's search of "rough" to provide the same results as "rou" so I'm not sure this is a route we want to take.

You can to some extent control this by setting `minimum_should_match`. If a search string generates multiple tokens you can specify that you want a portion of these to match for it to be successful. This would result in a different search result for `rou` and `rough` search strings.

---

<div class="post-metadata">

**Author:** ![cawoodm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cawoodm/32/14083_2.png) [@cawoodm](https://discuss.elastic.co/u/cawoodm)\
**Post date:** [October 3, 2023, 7:47am UTC](https://discuss.elastic.co/t/setting-min-max-in-ngram-tokenizers/344218/5 "2023-10-03T07:47:48Z")

</div>

```json
{
  ...
  "mappings": {
    "dynamic": "strict",
    "properties": {
     ...
      "fulltext": {
        "properties": {
          "de": {
            "analyzer": "ngram2",
            "search_analyzer": "whitesp",
            "type": "text"
          },
          "en": {
            "analyzer": "ngram2",
            "search_analyzer": "whitesp",
            "type": "text"
          }
        }
      ....
    }
  },
  "settings": {
    "index": {
      "analysis": {
        "analyzer": {
          "ngram2": {
            "filter": [
              "ngram2",
              "lowercase"
            ],
            "tokenizer": "whitespace"
          },
          "whitesp": {
            "filter": [
              "lowercase"
            ],
            "tokenizer": "whitespace"
          }
        },
        "filter": {
          "ngram2": {
            "max_gram": "20",
            "min_gram": "2",
            "type": "ngram"
          }
        }
      },
      "number_of_replicas": "1",
      "number_of_shards": "1",
      "max_ngram_diff": 18
    }
  }
}

```

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [October 3, 2023, 9:09am UTC](https://discuss.elastic.co/t/setting-min-max-in-ngram-tokenizers/344218/6 "2023-10-03T09:09:05Z")

</div>

> [@cawoodm](#):
>
> `"search_analyzer": "whitesp",`

You have a separate search analyser that does not generate ngrams, which explains why you do not get any match. If you remove this you will get a match as the ngram analyser will be used for both indexing and searching.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 31, 2023, 9:09am UTC](https://discuss.elastic.co/t/setting-min-max-in-ngram-tokenizers/344218/7 "2023-10-31T09:09:11Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.