# Tokenizer to get combinations of words

**URL:** <https://discuss.elastic.co/t/tokenizer-to-get-combinations-of-words/151396>\
**Category:** Elasticsearch\
**Created:** [October 8, 2018, 7:59am UTC](https://discuss.elastic.co/t/tokenizer-to-get-combinations-of-words/151396 "2018-10-08T07:59:46Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![Keerthika\_Raj](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/keerthika_raj/32/49152_2.png) [@Keerthika\_Raj](https://discuss.elastic.co/u/Keerthika_Raj)\
**Post date:** [October 8, 2018, 7:59am UTC](https://discuss.elastic.co/t/tokenizer-to-get-combinations-of-words/151396/1 "2018-10-08T07:59:46Z")

</div>

I have an index with a field that uses N-gram tokenizer to provide partial search across documents. I have a requirement where if the length of text to be indexed is more than threshold limit (say 300) i want to index it as a different property in the same index, that is to not use N-gram tokenizer, but still be able to support full text and phrase queries.

I want to search across both the fields. The problem i landed up is picking up the right tokenizer to index the content that is more than threshold length. The tokenizer i am looking for should split text as following.

```
"text":"The quick brown fox"
"tokens" : "The , quick, brown, fox, The quick, quick brown, brown fox, The quick brown, quick brown fox, The quick brown fox"

```

I have developed the following index mappings and settings and looking for a tokenizer for `value-meta` field.

```
{
 "mappings": {
 "message": {
  "properties": {
    "title": {
      "analyzer": "ngram-index_analyzer",
      "fields": {
        "keyword": {
          "type": "keyword"
        }
      },
      "search_analyzer": "search_analyzer",
      "type": "text"
    },
    "value": {
      "analyzer": "ngram-index_analyzer",
      "fields": {
        "keyword": {
          "type": "keyword"
        }
      },
      "search_analyzer": "search_analyzer",
      "type": "text"
    },
    "value-meta": {
      "fields": {
        "keyword": {
          "type": "keyword"
        }
      },         
      "type": "text"
    }
  }
}
},
"settings": {
 "analysis": {
  "analyzer": {
    "ngram-index_analyzer": {
      "filter": [
        "lowercase"
      ],
      "tokenizer": "ngram-index_analyzer"
    },
    "search_analyzer": {
      "filter": [
        "lowercase"
      ],
      "tokenizer": "keyword"
    }
  },
  "tokenizer": {
    "ngram-index_analyzer": {
      "max_gram": "30",
      "min_gram": "2",
      "token_chars": [
        "whitespace",
        "digit",
        "letter",
        "punctuation",
        "symbol"
      ],
      "type": "ngram"
    }
  }
},
"number_of_shards": "20"
}
}
```

---

<div class="post-metadata">

**Author:** ![Keerthika\_Raj](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/keerthika_raj/32/49152_2.png) [@Keerthika\_Raj](https://discuss.elastic.co/u/Keerthika_Raj)\
**Post date:** [October 17, 2018, 10:00am UTC](https://discuss.elastic.co/t/tokenizer-to-get-combinations-of-words/151396/2 "2018-10-17T10:00:41Z")

</div>

I achieved this by using whitespace analyzer as index and search analyzer for `value-meta` field.

.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [November 14, 2018, 10:00am UTC](https://discuss.elastic.co/t/tokenizer-to-get-combinations-of-words/151396/3 "2018-11-14T10:00:43Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.