# nGram filter and relevance score

**URL:** <https://discuss.elastic.co/t/ngram-filter-and-relevance-score/22475>\
**Category:** Elasticsearch\
**Created:** [March 2, 2015, 7:10pm UTC](https://discuss.elastic.co/t/ngram-filter-and-relevance-score/22475 "2015-03-02T19:10:26Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![Torben](https://avatars.discourse-cdn.com/v4/letter/t/5daacb/32.png) [@Torben](https://discuss.elastic.co/u/Torben)\
**Post date:** [March 2, 2015, 7:10pm UTC](https://discuss.elastic.co/t/ngram-filter-and-relevance-score/22475/1 "2015-03-02T19:10:26Z")

</div>

Hi everyone,

I'm using nGram filter for partial matching and have some problems with  
relevance scoring in my search results.

With multi\_field and the standard analyzer I can boost the exact match e.g.  
"foo", which is good.  
But I also want the term "barfoobar" to have a higher score than "  
blablablafoobarbarbar", because the field length is shorter.  
I thought this is the default in calculating the relevance score, but in  
fact they both have the same score (0.36101705).

How can I implement this?

Full demo here:

# create index

curl -XPUT 'localhost:9200/test' -d'  
{  
"settings": {  
"number\_of\_shards": 1,  
"number\_of\_replicas": 0,  
"analysis": {  
"analyzer": {  
"my\_index\_analyzer": {  
"type": "custom",  
"tokenizer": "whitespace",  
"filter": [  
"lowercase",  
"my\_nGram"  
]  
},  
"my\_search\_analyzer": {  
"type": "custom",  
"tokenizer": "whitespace",  
"filter": [  
"lowercase"  
]  
}  
},  
"filter": {  
"my\_nGram": {  
"type": "nGram",  
"min\_gram": 1,  
"max\_gram": 50  
}  
}  
}  
},  
"mappings": {  
"my\_type": {  
"\_all": {  
"enabled": false  
},  
"properties": {  
"my\_field": {  
"type": "multi\_field",  
"fields": {  
"my\_field": {  
"type": "string",  
"index\_analyzer": "standard",  
"search\_analyzer": "standard",  
"include\_in\_all": false,  
"boost": 10.0  
},  
"partial": {  
"type": "string",  
"index\_analyzer": "my\_index\_analyzer",  
"search\_analyzer": "my\_search\_analyzer",  
"include\_in\_all": false,  
"boost": 1.0  
}  
}  
}  
}  
}  
}  
}  
'

# import data

curl -XPUT 'localhost:9200/test/my\_type/1' -d '  
{  
"my\_field": "blablablafoobarbarbar"  
}  
'

curl -XPUT 'localhost:9200/test/my\_type/2' -d '  
{  
"my\_field": "foo"  
}  
'

curl -XPUT 'localhost:9200/test/my\_type/3' -d '  
{  
"my\_field": "barfoobar"  
}  
'

# search

curl -XGET 'localhost:9200/test/my\_type/\_search?pretty' -d '  
{  
"query": {  
"multi\_match": {  
"query": "foo",  
"fields": [  
"my\_field",  
"my\_field.partial"  
],  
"operator": "and"  
}  
}  
}  
'

# result

{  
"took" : 1,  
"timed\_out" : false,  
"\_shards" : {  
"total" : 1,  
"successful" : 1,  
"failed" : 0  
},  
"hits" : {  
"total" : 3,  
"max\_score" : 14.054651,  
"hits" : [ {  
"\_index" : "test",  
"\_type" : "my\_type",  
"\_id" : "2",  
"\_score" : 14.054651,  
"\_source":  
{  
"my\_field": "foo"  
}

```
}, {
  "_index" : "test",
  "_type" : "my_type",
  "_id" : "1",
  "_score" : 0.36101705,
  "_source":

```

{  
"my\_field": "blablablafoobarbarbar"  
}

```
}, {
  "_index" : "test",
  "_type" : "my_type",
  "_id" : "3",
  "_score" : 0.36101705,
  "_source":

```

{  
"my\_field": "barfoobar"  
}

```
} ]

```

}  
}

Thanks in advance and best regards,  
Torben

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/958585db-8c0a-4c9b-86b2-3df9513b5a7f%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/958585db-8c0a-4c9b-86b2-3df9513b5a7f%40googlegroups.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![Torben](https://avatars.discourse-cdn.com/v4/letter/t/5daacb/32.png) [@Torben](https://discuss.elastic.co/u/Torben)\
**Post date:** [March 5, 2015, 8:56pm UTC](https://discuss.elastic.co/t/ngram-filter-and-relevance-score/22475/2 "2015-03-05T20:56:40Z")

</div>

If anyone is interested:

I resolved this issue by replacing the nGram filter with a nGram tokenizer:

{  
"settings": {  
"number\_of\_shards": 1,  
"number\_of\_replicas": 0,  
"analysis": {  
"analyzer": {  
"my\_index\_analyzer": {  
"type": "custom",  
"tokenizer": "my\_ngram\_tokenizer",  
"filter": [  
"lowercase"  
]  
},  
"my\_search\_analyzer": {  
"type": "custom",  
"tokenizer": "whitespace",  
"filter": [  
"lowercase"  
]  
}  
},  
"tokenizer" : {  
"my\_ngram\_tokenizer" : {  
"type" : "nGram",  
"min\_gram" : "1",  
"max\_gram" : "50"  
}  
}  
}  
}  
}

Now, the term "barfoobar" has a higher score than "blablablafoobarbarbar".

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/521b01f1-c33f-4444-a618-edb458cd6717%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/521b01f1-c33f-4444-a618-edb458cd6717%40googlegroups.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![jpountz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jpountz/32/45836_2.png) [@jpountz](https://discuss.elastic.co/u/jpountz)\
**Post date:** [March 6, 2015, 8:39am UTC](https://discuss.elastic.co/t/ngram-filter-and-relevance-score/22475/3 "2015-03-06T08:39:49Z")

</div>

Hi Torben,

Indeed, this is due to the fact that the ngram FILTER writes terms at the  
same position (like synonyms) while the TOKENIZER generates a stream of  
tokens which have consecutive positions. This gives blablablafoobarbarbar a  
larger number of positions and thus a smaller length normalization.

On Thu, Mar 5, 2015 at 9:56 PM, Torben [can.i.hazz@gmail.com](mailto:can.i.hazz@gmail.com) wrote:

> If anyone is interested:
> 
> I resolved this issue by replacing the nGram filter with a nGram tokenizer:
> 
> {  
> "settings": {  
> "number\_of\_shards": 1,  
> "number\_of\_replicas": 0,  
> "analysis": {  
> "analyzer": {  
> "my\_index\_analyzer": {  
> "type": "custom",  
> "tokenizer": "my\_ngram\_tokenizer",  
> "filter": [  
> "lowercase"  
> ]  
> },  
> "my\_search\_analyzer": {  
> "type": "custom",  
> "tokenizer": "whitespace",  
> "filter": [  
> "lowercase"  
> ]  
> }  
> },  
> "tokenizer" : {  
> "my\_ngram\_tokenizer" : {  
> "type" : "nGram",  
> "min\_gram" : "1",  
> "max\_gram" : "50"  
> }  
> }  
> }  
> }  
> }
> 
> Now, the term "barfoobar" has a higher score than "blablablafoobarbarbar".
> 
> --  
> You received this message because you are subscribed to the Google Groups  
> "elasticsearch" group.  
> To unsubscribe from this group and stop receiving emails from it, send an  
> email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
> To view this discussion on the web visit  
> [https://groups.google.com/d/msgid/elasticsearch/521b01f1-c33f-4444-a618-edb458cd6717%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/521b01f1-c33f-4444-a618-edb458cd6717%40googlegroups.com)  
> [https://groups.google.com/d/msgid/elasticsearch/521b01f1-c33f-4444-a618-edb458cd6717%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/521b01f1-c33f-4444-a618-edb458cd6717%40googlegroups.com?utm_medium=email&utm_source=footer)  
> .
> 
> For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

--  
Adrien Grand

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j5pz5UNZAXaFHsCtjqsLLnuLfn\_Xp3tq4xaotiU0mMNag%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j5pz5UNZAXaFHsCtjqsLLnuLfn_Xp3tq4xaotiU0mMNag%40mail.gmail.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 12:28am UTC](https://discuss.elastic.co/t/ngram-filter-and-relevance-score/22475/4 "2017-07-06T00:28:08Z")

</div>