# Analyzer change in behaviour in 0.16 - bug? feature?

**URL:** <https://discuss.elastic.co/t/analyzer-change-in-behaviour-in-0-16-bug-feature/4296>\
**Category:** Elasticsearch\
**Created:** [April 26, 2011, 3:04pm UTC](https://discuss.elastic.co/t/analyzer-change-in-behaviour-in-0-16-bug-feature/4296 "2011-04-26T15:04:11Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![Clinton\_Gormley](https://avatars.discourse-cdn.com/v4/letter/c/50afbb/32.png) [@Clinton\_Gormley](https://discuss.elastic.co/u/Clinton_Gormley)\
**Post date:** [April 26, 2011, 3:04pm UTC](https://discuss.elastic.co/t/analyzer-change-in-behaviour-in-0-16-bug-feature/4296/1 "2011-04-26T15:04:11Z")

</div>

Hiya

There has been a change in behaviour in how analyzers are applied  
between 0.15.2 and 0.16.0.

For instance, in 0.15.2, an 'edge-ngram' tokenizer, was being applied at  
index time, but not at search time.

In 0.16.0, it is also applied at search time.

For instance, I index a field containing "london" (with an edge-ngram  
tokenizer). I expect a search for 'lon' to match the doc, but not a  
search for 'londres'. This was the case in 0.15

However, because the query term is being passed through the same  
tokenizer, the query is actually for the terms  
"l","lo","lon","lond",londr"...etc so this search DOES match in 0.16

It can easily be worked around by using a different analyzer at index  
and search time, but I'm not sure that this is correct behaviour. It  
makes sense for (eg) snowball analyzers, but ngrams?

What do you think?

clint

curl -XPUT '[http://127.0.0.1:9200/foo/?pretty=1](http://127.0.0.1:9200/foo/?pretty=1)' -d '  
{  
"mappings" : {  
"bar" : {  
"properties" : {  
"tokens" : {  
"type" : "string",  
"analyzer" : "edge\_ngram"  
}  
}  
}  
},  
"settings" : {  
"analysis" : {  
"filter" : {  
"edge\_ngram" : {  
"side" : "front",  
"max\_gram" : 20,  
"min\_gram" : 1,  
"type" : "edgeNGram"  
}  
},  
"analyzer" : {  
"edge\_ngram" : {  
"filter" : [  
"standard",  
"lowercase",  
"edge\_ngram"  
],  
"type" : "custom",  
"tokenizer" : "standard"  
}  
}  
}  
}  
}  
'

curl -XPOST '[http://127.0.0.1:9200/foo/bar?pretty=1&refresh=true](http://127.0.0.1:9200/foo/bar?pretty=1&refresh=true)' -d '  
{  
"tokens" : "london"  
}  
'

curl -XGET '[http://127.0.0.1:9200/foo/bar/\_search?pretty=1](http://127.0.0.1:9200/foo/bar/_search?pretty=1)' -d '  
{  
"query" : {  
"field" : {  
"tokens" : "londres"  
}  
}  
}  
'

RESULT IN 0.15.2:

# {

# "hits" : {

# "hits" : [],

# "max\_score" : null,

# "total" : 0

# },

# "timed\_out" : false,

# "\_shards" : {

# "failed" : 0,

# "successful" : 5,

# "total" : 5

# },

# "took" : 3

# }

RESULT IN 0.16.0:

# [Tue Apr 26 17:02:37 2011] Response:

# {

# "hits" : {

# "hits" : [

# {

# "\_source" : {

# "tokens" : "london"

# },

# "\_score" : 0.043920923,

# "\_index" : "foo",

# "\_id" : "M0aZ9of6Q7e1gEOe7syQvA",

# "\_type" : "bar"

# }

# ],

# "max\_score" : 0.043920923,

# "total" : 1

# },

# "timed\_out" : false,

# "\_shards" : {

# "failed" : 0,

# "successful" : 5,

# "total" : 5

# },

# "took" : 51

# }

---

<div class="post-metadata">

**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\
**Post date:** [April 26, 2011, 5:37pm UTC](https://discuss.elastic.co/t/analyzer-change-in-behaviour-in-0-16-bug-feature/4296/2 "2011-04-26T17:37:10Z")

</div>

I need to verify and track the what you see, but, when an analyzer is set in the mappings, it is applied both for search and query time (for query\_string / field queries), that is the intended behavior, regardless of what type of analyzer you use.  
On Tuesday, April 26, 2011 at 6:04 PM, Clinton Gormley wrote:

> Hiya
> 
> There has been a change in behaviour in how analyzers are applied  
> between 0.15.2 and 0.16.0.
> 
> For instance, in 0.15.2, an 'edge-ngram' tokenizer, was being applied at  
> index time, but not at search time.
> 
> In 0.16.0, it is also applied at search time.
> 
> For instance, I index a field containing "london" (with an edge-ngram  
> tokenizer). I expect a search for 'lon' to match the doc, but not a  
> search for 'londres'. This was the case in 0.15
> 
> However, because the query term is being passed through the same  
> tokenizer, the query is actually for the terms  
> "l","lo","lon","lond",londr"...etc so this search DOES match in 0.16
> 
> It can easily be worked around by using a different analyzer at index  
> and search time, but I'm not sure that this is correct behaviour. It  
> makes sense for (eg) snowball analyzers, but ngrams?
> 
> What do you think?
> 
> clint
> 
> curl -XPUT '[http://127.0.0.1:9200/foo/?pretty=1](http://127.0.0.1:9200/foo/?pretty=1)' -d '  
> {  
> "mappings" : {  
> "bar" : {  
> "properties" : {  
> "tokens" : {  
> "type" : "string",  
> "analyzer" : "edge\_ngram"  
> }  
> }  
> }  
> },  
> "settings" : {  
> "analysis" : {  
> "filter" : {  
> "edge\_ngram" : {  
> "side" : "front",  
> "max\_gram" : 20,  
> "min\_gram" : 1,  
> "type" : "edgeNGram"  
> }  
> },  
> "analyzer" : {  
> "edge\_ngram" : {  
> "filter" : [  
> "standard",  
> "lowercase",  
> "edge\_ngram"  
> ],  
> "type" : "custom",  
> "tokenizer" : "standard"  
> }  
> }  
> }  
> }  
> }  
> '
> 
> curl -XPOST '[http://127.0.0.1:9200/foo/bar?pretty=1&refresh=true](http://127.0.0.1:9200/foo/bar?pretty=1&refresh=true)' -d '  
> {  
> "tokens" : "london"  
> }  
> '
> 
> curl -XGET '[http://127.0.0.1:9200/foo/bar/\_search?pretty=1](http://127.0.0.1:9200/foo/bar/_search?pretty=1)' -d '  
> {  
> "query" : {  
> "field" : {  
> "tokens" : "londres"  
> }  
> }  
> }  
> '
> 
> RESULT IN 0.15.2:
> 
> # {
> 
> # "hits" : {
> 
> # "hits" : ,
> 
> # "max\_score" : null,
> 
> # "total" : 0
> 
> # },
> 
> # "timed\_out" : false,
> 
> # "\_shards" : {
> 
> # "failed" : 0,
> 
> # "successful" : 5,
> 
> # "total" : 5
> 
> # },
> 
> # "took" : 3
> 
> # }
> 
> RESULT IN 0.16.0:
> 
> # [Tue Apr 26 17:02:37 2011] Response:
> 
> # {
> 
> # "hits" : {
> 
> # "hits" : [
> 
> # {
> 
> # "\_source" : {
> 
> # "tokens" : "london"
> 
> # },
> 
> # "\_score" : 0.043920923,
> 
> # "\_index" : "foo",
> 
> # "\_id" : "M0aZ9of6Q7e1gEOe7syQvA",
> 
> # "\_type" : "bar"
> 
> # }
> 
> # ],
> 
> # "max\_score" : 0.043920923,
> 
> # "total" : 1
> 
> # },
> 
> # "timed\_out" : false,
> 
> # "\_shards" : {
> 
> # "failed" : 0,
> 
> # "successful" : 5,
> 
> # "total" : 5
> 
> # },
> 
> # "took" : 51
> 
> # }

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 4:07am UTC](https://discuss.elastic.co/t/analyzer-change-in-behaviour-in-0-16-bug-feature/4296/3 "2017-07-06T04:07:38Z")

</div>