### Author:  [@Jonathan\_Evans](https://discuss.elastic.co/u/Jonathan_Evans)
#### Post date: [January 15, 2013, 8:09am UTC](https://discuss.elastic.co/t/hypenation-and-superfluous-results-with-ngram-analyser-for-autocomplete/10346/1 "2013-01-15T08:09:58Z")
I am trying to configure elasticsearch for autocomplete and have been
quite successful in doing so, however there are a couple of behaviours I
would like to tweak if possible.
1. When searching for 'Mercedes-Benz' no results are returned with the
current setup even though one of the indexed items contains the term.
'mercedes benz' 'merc' and 'benz' all match the right item as expected.
2. When searching for 'Mercedes-Be' I get a superfluous result: "Being
Cool With Bond, James Bond". The term is obviously being broken into
'mercedes' and 'be', the latter matching the start of "Being" however I
would rather the second word act to further limit the results presented to
the user (as is probably expected).
The results, settings and mapping are listed in the following gist:
>
Could anyone offer any guidance on how to fix these issues?
Cheers,
Jon
--
---
### Author:  [@Igor\_Motov](https://discuss.elastic.co/u/Igor_Motov)
#### Post date: [January 15, 2013, 4:40pm UTC](https://discuss.elastic.co/t/hypenation-and-superfluous-results-with-ngram-analyser-for-autocomplete/10346/2 "2013-01-15T16:40:39Z")
During indexing you are using standard tokenizer that splits words on "-".
So, 'Mercedes-Benz' is indexed like this:
$ curl -s
"localhost:9200/courses/\_analyze?analyzer=autocomplete\_analyzer&pretty=true"
-d "Mercedes-Benz" | grep "token"
"token" : "me",
"token" : "mer",
"token" : "merc",
"token" : "merce",
"token" : "merced",
"token" : "mercede",
"token" : "mercedes",
"token" : "be",
"token" : "ben",
"token" : "benz",
For the search part you are using keyword tokenizer, which doesn't tokenize
at all, so as a result the query for "mercedes-benz" is getting translated
into the query for the term "mercedes-benz":
$ curl -s -X GET
'[http://localhost:9200/courses/course/\_validate/query?pretty=true&explain=true](http://localhost:9200/courses/course/_validate/query?pretty=true&explain=true)'
-d '{
"query\_string":{
"query":"mercedes-benz",
"fields":[
"name"
]
}
}' | grep "explanation"
_"explanation" : "name:mercedes-benz"_
There is no token "mercedes-benz" in the index, so you get no results. When
you replace "-" with space, the query\_string parser splits the query into
to parts:
$ curl -s -X GET
'[http://localhost:9200/courses/course/\_validate/query?pretty=true&explain=true](http://localhost:9200/courses/course/_validate/query?pretty=true&explain=true)'
-d '{
"query\_string":{
"query":"mercedes benz",
"fields":[
"name"
]
}
}' | grep "explanation"
_"explanation" : "name:mercedes name:benz"_
It searches for the term mercedes OR the term benz and you get expected
result. But because of this "OR" you are also finding "Being Cool..." when
you replace your query with "mercedes be".
To fix it, you should first replace query\_string query with something that
wouldn't interfere with your tokenization. The Match[http://www.elasticsearch.org/guide/reference/query-dsl/match-query.html](http://www.elasticsearch.org/guide/reference/query-dsl/match-query.html)query might be a good candidate. It still leaves the mismatch between
search tokenizer and index tokenizer to be addressed. There are few options
here. The simplest one is to replace the search analyzer with standard
analyzer with no stop words. That will work work in most case. The only
potential issue here is that it will disregard word order in you search. So
it will also find you "Being Cool" when you search for "Cool Being". Not
sure if this is something that you want to avoid or not.
On Tuesday, January 15, 2013 3:09:58 AM UTC-5, Jonathan Evans wrote:
> I am trying to configure elasticsearch for autocomplete and have been
> quite successful in doing so, however there are a couple of behaviours I
> would like to tweak if possible.
>
> 1. When searching for 'Mercedes-Benz' no results are returned with the
> current setup even though one of the indexed items contains the term.
> 'mercedes benz' 'merc' and 'benz' all match the right item as expected.
>
> 2. When searching for 'Mercedes-Be' I get a superfluous result: "Being
> Cool With Bond, James Bond". The term is obviously being broken into
> 'mercedes' and 'be', the latter matching the start of "Being" however I
> would rather the second word act to further limit the results presented to
> the user (as is probably expected).
>
> The results, settings and mapping are listed in the following gist:
> [Demonstrates two unwanted results with current elasticsearch setup. · GitHub](https://gist.github.com/4537084)
>
> Could anyone offer any guidance on how to fix these issues?
>
> Cheers,
>
> Jon
--
---
### Author:  [@Jonathan\_Evans](https://discuss.elastic.co/u/Jonathan_Evans)
#### Post date: [January 16, 2013, 10:48am UTC](https://discuss.elastic.co/t/hypenation-and-superfluous-results-with-ngram-analyser-for-autocomplete/10346/3 "2013-01-16T10:48:30Z")
Cheers Igor, this works great! I have a much better understanding after
your detailed breakdown as well.
One further undesirable thing I have noticed is that if I type Mercedes
Benz Drama it returns an item "primary drama". I cannot think of anyway
really to fix this as its the same sort of problem as the ordering problem
you mentioned.
Any ideas?
On Tuesday, January 15, 2013 4:40:39 PM UTC, Igor Motov wrote:
> During indexing you are using standard tokenizer that splits words on "-".
> So, 'Mercedes-Benz' is indexed like this:
>
> $ curl -s
> "localhost:9200/courses/\_analyze?analyzer=autocomplete\_analyzer&pretty=true"
> -d "Mercedes-Benz" | grep "token"
> "token" : "me",
> "token" : "mer",
> "token" : "merc",
> "token" : "merce",
> "token" : "merced",
> "token" : "mercede",
> "token" : "mercedes",
> "token" : "be",
> "token" : "ben",
> "token" : "benz",
>
> For the search part you are using keyword tokenizer, which doesn't
> tokenize at all, so as a result the query for "mercedes-benz" is getting
> translated into the query for the term "mercedes-benz":
>
> $ curl -s -X GET '
> [http://localhost:9200/courses/course/\_validate/query?pretty=true&explain=true](http://localhost:9200/courses/course/_validate/query?pretty=true&explain=true)'
> -d '{
> "query\_string":{
> "query":"mercedes-benz",
> "fields":[
> "name"
> ]
> }
> }' | grep "explanation"
> _"explanation" : "name:mercedes-benz"_
>
> There is no token "mercedes-benz" in the index, so you get no results.
> When you replace "-" with space, the query\_string parser splits the query
> into to parts:
>
> $ curl -s -X GET '
> [http://localhost:9200/courses/course/\_validate/query?pretty=true&explain=true](http://localhost:9200/courses/course/_validate/query?pretty=true&explain=true)'
> -d '{
> "query\_string":{
> "query":"mercedes benz",
> "fields":[
> "name"
> ]
> }
> }' | grep "explanation"
> _"explanation" : "name:mercedes name:benz"_
>
> It searches for the term mercedes OR the term benz and you get expected
> result. But because of this "OR" you are also finding "Being Cool..." when
> you replace your query with "mercedes be".
>
> To fix it, you should first replace query\_string query with something that
> wouldn't interfere with your tokenization. The Match[http://www.elasticsearch.org/guide/reference/query-dsl/match-query.html](http://www.elasticsearch.org/guide/reference/query-dsl/match-query.html)query might be a good candidate. It still leaves the mismatch between
> search tokenizer and index tokenizer to be addressed. There are few options
> here. The simplest one is to replace the search analyzer with standard
> analyzer with no stop words. That will work work in most case. The only
> potential issue here is that it will disregard word order in you search. So
> it will also find you "Being Cool" when you search for "Cool Being". Not
> sure if this is something that you want to avoid or not.
>
> On Tuesday, January 15, 2013 3:09:58 AM UTC-5, Jonathan Evans wrote:
>
> > I am trying to configure elasticsearch for autocomplete and have been
> > quite successful in doing so, however there are a couple of behaviours I
> > would like to tweak if possible.
> >
> > 1. When searching for 'Mercedes-Benz' no results are returned with the
> > current setup even though one of the indexed items contains the term.
> > 'mercedes benz' 'merc' and 'benz' all match the right item as expected.
> >
> > 2. When searching for 'Mercedes-Be' I get a superfluous result: "Being
> > Cool With Bond, James Bond". The term is obviously being broken into
> > 'mercedes' and 'be', the latter matching the start of "Being" however I
> > would rather the second word act to further limit the results presented to
> > the user (as is probably expected).
> >
> > The results, settings and mapping are listed in the following gist:
> > [Demonstrates two unwanted results with current elasticsearch setup. · GitHub](https://gist.github.com/4537084)
> >
> > Could anyone offer any guidance on how to fix these issues?
> >
> > Cheers,
> >
> > Jon
--
---
### Author:  [@polyfractal](https://discuss.elastic.co/u/polyfractal)
#### Post date: [January 16, 2013, 12:45pm UTC](https://discuss.elastic.co/t/hypenation-and-superfluous-results-with-ngram-analyser-for-autocomplete/10346/4 "2013-01-16T12:45:04Z")
You could add an additional query like text\_phrase that boosts phrase
matches. Alternatively, you could build phrase matches into your index
with something like shingles.
-Zach
On Wednesday, January 16, 2013 5:48:30 AM UTC-5, Jonathan Evans wrote:
> Cheers Igor, this works great! I have a much better understanding after
> your detailed breakdown as well.
>
> One further undesirable thing I have noticed is that if I type Mercedes
> Benz Drama it returns an item "primary drama". I cannot think of anyway
> really to fix this as its the same sort of problem as the ordering problem
> you mentioned.
>
> Any ideas?
>
> On Tuesday, January 15, 2013 4:40:39 PM UTC, Igor Motov wrote:
>
> > During indexing you are using standard tokenizer that splits words on
> > "-". So, 'Mercedes-Benz' is indexed like this:
> >
> > $ curl -s
> > "localhost:9200/courses/\_analyze?analyzer=autocomplete\_analyzer&pretty=true"
> > -d "Mercedes-Benz" | grep "token"
> > "token" : "me",
> > "token" : "mer",
> > "token" : "merc",
> > "token" : "merce",
> > "token" : "merced",
> > "token" : "mercede",
> > "token" : "mercedes",
> > "token" : "be",
> > "token" : "ben",
> > "token" : "benz",
> >
> > For the search part you are using keyword tokenizer, which doesn't
> > tokenize at all, so as a result the query for "mercedes-benz" is getting
> > translated into the query for the term "mercedes-benz":
> >
> > $ curl -s -X GET '
> > [http://localhost:9200/courses/course/\_validate/query?pretty=true&explain=true](http://localhost:9200/courses/course/_validate/query?pretty=true&explain=true)'
> > -d '{
> > "query\_string":{
> > "query":"mercedes-benz",
> > "fields":[
> > "name"
> > ]
> > }
> > }' | grep "explanation"
> > _"explanation" : "name:mercedes-benz"_
> >
> > There is no token "mercedes-benz" in the index, so you get no results.
> > When you replace "-" with space, the query\_string parser splits the query
> > into to parts:
> >
> > $ curl -s -X GET '
> > [http://localhost:9200/courses/course/\_validate/query?pretty=true&explain=true](http://localhost:9200/courses/course/_validate/query?pretty=true&explain=true)'
> > -d '{
> > "query\_string":{
> > "query":"mercedes benz",
> > "fields":[
> > "name"
> > ]
> > }
> > }' | grep "explanation"
> > _"explanation" : "name:mercedes name:benz"_
> >
> > It searches for the term mercedes OR the term benz and you get expected
> > result. But because of this "OR" you are also finding "Being Cool..." when
> > you replace your query with "mercedes be".
> >
> > To fix it, you should first replace query\_string query with something
> > that wouldn't interfere with your tokenization. The Match[http://www.elasticsearch.org/guide/reference/query-dsl/match-query.html](http://www.elasticsearch.org/guide/reference/query-dsl/match-query.html)query might be a good candidate. It still leaves the mismatch between
> > search tokenizer and index tokenizer to be addressed. There are few options
> > here. The simplest one is to replace the search analyzer with standard
> > analyzer with no stop words. That will work work in most case. The only
> > potential issue here is that it will disregard word order in you search. So
> > it will also find you "Being Cool" when you search for "Cool Being". Not
> > sure if this is something that you want to avoid or not.
> >
> > On Tuesday, January 15, 2013 3:09:58 AM UTC-5, Jonathan Evans wrote:
> >
> > > I am trying to configure elasticsearch for autocomplete and have been
> > > quite successful in doing so, however there are a couple of behaviours I
> > > would like to tweak if possible.
> > >
> > > 1. When searching for 'Mercedes-Benz' no results are returned with the
> > > current setup even though one of the indexed items contains the term.
> > > 'mercedes benz' 'merc' and 'benz' all match the right item as expected.
> > >
> > > 2. When searching for 'Mercedes-Be' I get a superfluous result: "Being
> > > Cool With Bond, James Bond". The term is obviously being broken into
> > > 'mercedes' and 'be', the latter matching the start of "Being" however I
> > > would rather the second word act to further limit the results presented to
> > > the user (as is probably expected).
> > >
> > > The results, settings and mapping are listed in the following gist:
> > > [Demonstrates two unwanted results with current elasticsearch setup. · GitHub](https://gist.github.com/4537084)
> > >
> > > Could anyone offer any guidance on how to fix these issues?
> > >
> > > Cheers,
> > >
> > > Jon
--
---
### Author:  [@Igor\_Motov](https://discuss.elastic.co/u/Igor_Motov)
#### Post date: [January 16, 2013, 1:54pm UTC](https://discuss.elastic.co/t/hypenation-and-superfluous-results-with-ngram-analyser-for-autocomplete/10346/5 "2013-01-16T13:54:06Z")
It happens because match is using OR operator by default as well. But
that's easy to fix. Just change operator to "and" in your match query like
this:
```
"match":{
"name":{
"query": "Mercedes-Benz",
```
-
```
"operator": "and"*
}
```
}
On Wednesday, January 16, 2013 5:48:30 AM UTC-5, Jonathan Evans wrote:
> Cheers Igor, this works great! I have a much better understanding after
> your detailed breakdown as well.
>
> One further undesirable thing I have noticed is that if I type Mercedes
> Benz Drama it returns an item "primary drama". I cannot think of anyway
> really to fix this as its the same sort of problem as the ordering problem
> you mentioned.
>
> Any ideas?
>
> On Tuesday, January 15, 2013 4:40:39 PM UTC, Igor Motov wrote:
>
> > During indexing you are using standard tokenizer that splits words on
> > "-". So, 'Mercedes-Benz' is indexed like this:
> >
> > $ curl -s
> > "localhost:9200/courses/\_analyze?analyzer=autocomplete\_analyzer&pretty=true"
> > -d "Mercedes-Benz" | grep "token"
> > "token" : "me",
> > "token" : "mer",
> > "token" : "merc",
> > "token" : "merce",
> > "token" : "merced",
> > "token" : "mercede",
> > "token" : "mercedes",
> > "token" : "be",
> > "token" : "ben",
> > "token" : "benz",
> >
> > For the search part you are using keyword tokenizer, which doesn't
> > tokenize at all, so as a result the query for "mercedes-benz" is getting
> > translated into the query for the term "mercedes-benz":
> >
> > $ curl -s -X GET '
> > [http://localhost:9200/courses/course/\_validate/query?pretty=true&explain=true](http://localhost:9200/courses/course/_validate/query?pretty=true&explain=true)'
> > -d '{
> > "query\_string":{
> > "query":"mercedes-benz",
> > "fields":[
> > "name"
> > ]
> > }
> > }' | grep "explanation"
> > _"explanation" : "name:mercedes-benz"_
> >
> > There is no token "mercedes-benz" in the index, so you get no results.
> > When you replace "-" with space, the query\_string parser splits the query
> > into to parts:
> >
> > $ curl -s -X GET '
> > [http://localhost:9200/courses/course/\_validate/query?pretty=true&explain=true](http://localhost:9200/courses/course/_validate/query?pretty=true&explain=true)'
> > -d '{
> > "query\_string":{
> > "query":"mercedes benz",
> > "fields":[
> > "name"
> > ]
> > }
> > }' | grep "explanation"
> > _"explanation" : "name:mercedes name:benz"_
> >
> > It searches for the term mercedes OR the term benz and you get expected
> > result. But because of this "OR" you are also finding "Being Cool..." when
> > you replace your query with "mercedes be".
> >
> > To fix it, you should first replace query\_string query with something
> > that wouldn't interfere with your tokenization. The Match[http://www.elasticsearch.org/guide/reference/query-dsl/match-query.html](http://www.elasticsearch.org/guide/reference/query-dsl/match-query.html)query might be a good candidate. It still leaves the mismatch between
> > search tokenizer and index tokenizer to be addressed. There are few options
> > here. The simplest one is to replace the search analyzer with standard
> > analyzer with no stop words. That will work work in most case. The only
> > potential issue here is that it will disregard word order in you search. So
> > it will also find you "Being Cool" when you search for "Cool Being". Not
> > sure if this is something that you want to avoid or not.
> >
> > On Tuesday, January 15, 2013 3:09:58 AM UTC-5, Jonathan Evans wrote:
> >
> > > I am trying to configure elasticsearch for autocomplete and have been
> > > quite successful in doing so, however there are a couple of behaviours I
> > > would like to tweak if possible.
> > >
> > > 1. When searching for 'Mercedes-Benz' no results are returned with the
> > > current setup even though one of the indexed items contains the term.
> > > 'mercedes benz' 'merc' and 'benz' all match the right item as expected.
> > >
> > > 2. When searching for 'Mercedes-Be' I get a superfluous result: "Being
> > > Cool With Bond, James Bond". The term is obviously being broken into
> > > 'mercedes' and 'be', the latter matching the start of "Being" however I
> > > would rather the second word act to further limit the results presented to
> > > the user (as is probably expected).
> > >
> > > The results, settings and mapping are listed in the following gist:
> > > [Demonstrates two unwanted results with current elasticsearch setup. · GitHub](https://gist.github.com/4537084)
> > >
> > > Could anyone offer any guidance on how to fix these issues?
> > >
> > > Cheers,
> > >
> > > Jon
--
---
### Author:  [@btiernay](https://discuss.elastic.co/u/btiernay)
#### Post date: [January 17, 2013, 3:19am UTC](https://discuss.elastic.co/t/hypenation-and-superfluous-results-with-ngram-analyser-for-autocomplete/10346/6 "2013-01-17T03:19:12Z")
Awesome response!
--
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 6, 2017, 2:56am UTC](https://discuss.elastic.co/t/hypenation-and-superfluous-results-with-ngram-analyser-for-autocomplete/10346/7 "2017-07-06T02:56:01Z")