# nGram ordered partial/phrase matching - how to split search token by length?

**URL:** <https://discuss.elastic.co/t/ngram-ordered-partial-phrase-matching-how-to-split-search-token-by-length/152520>\
**Category:** Elasticsearch\
**Created:** [October 15, 2018, 2:57pm UTC](https://discuss.elastic.co/t/ngram-ordered-partial-phrase-matching-how-to-split-search-token-by-length/152520 "2018-10-15T14:57:43Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![tomekit](https://avatars.discourse-cdn.com/v4/letter/t/e5b9ba/32.png) [@tomekit](https://discuss.elastic.co/u/tomekit)\
**Post date:** [October 15, 2018, 2:57pm UTC](https://discuss.elastic.co/t/ngram-ordered-partial-phrase-matching-how-to-split-search-token-by-length/152520/1 "2018-10-15T14:57:43Z")

</div>

I am trying to implement autocomplete for chemical substance names, e.g. `1-Methyl-1,3,4,5-tetrahydropyrrolo(4,3,2-de)isoquinoline` where it's possible to search using partial name.  
E.g. `yrrolo(4,3,2`

I've got this working fine with `nGram` tokenizer, building nGrams from 1 to 10.  
Unfortunately this also needs to work for longer substrings, up to 50+ characters.  
Currently I can find substances using search term up to nGram max length (which is 10).  
I don't want to build nGrams up to 50 or more as it's time/space/resource consuming and I believe I can achieve it with mix of shorter nGrams and partial matching.

My idea is to use the: [https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-match-query-phrase.html](https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-match-query-phrase.html)  
against my nGram index.  
Since my nGrams are limited to 10 characters I need to split search phrase (using search analyzer) on chunks of up to 10 (maximum length of nGram).

I couldn't find any off-the-shelv tokenizer which can split phrase on tokens of specified length, hence my use of `pattern` tokenizer:

```
curl -XPUT "localhost:9200/my_index" -d'{
  "settings": {
    "analysis": {
      "analyzer": {
        "my_analyzer": {
          "tokenizer": "my_tokenizer",
          "char_filter": [],
          "filter": []
        }
      },
      "tokenizer": {
        "my_tokenizer": {
          "type": "pattern",
          "pattern": "(.{1,10})",
          "group": "1"
        }
      }
    }
  }
}'

```

Test link: [http://localhost:9200/my\_index/\_analyze?text=quitelongcomma,quitelongseparated,values&analyzer=my\_analyzer](http://localhost:9200/my_index/_analyze?text=quitelongcomma,quitelongseparated,values&analyzer=my_analyzer)

It looks promising, unfortunately I can't understand why the results from the above `_analyze` (in link) are split not only by token length (which is up to 10 characters or less if there is no more), but also on `comma` character.  
Results:  
`quitelongc`  
`omma`  
`quitelongs`  
`eparated`  
`values`

Expected:  
`quitelong,`  
`comma,quit`  
`elongsepar`  
`ated,value`  
`s`

If I use different character than `comma` in test analyze, e.g. `dot` then results are split correctly by token length, but not by this `dot` separator.  
How to skip comma, so it's not considered a stop character for pattern tokenizer?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [November 12, 2018, 2:57pm UTC](https://discuss.elastic.co/t/ngram-ordered-partial-phrase-matching-how-to-split-search-token-by-length/152520/2 "2018-11-12T14:57:48Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.