EdgeNgram tokenizer: Get only high quality hits?


(Karan Verma) #1

Hi

I implemented a custom filter which uses the EdgeGram tokenizer. The
problem I face is that whether I search for something relevant or total
garbage I get a large number of hits. I suspect that this is due that fact
that I'm using an EdgeNgram tokenizer. How can I ensure the quality of the
search results and have ES return no matches for low quality results?

Does the fact that elastic search returns a large number of hits (~100,000)
mean that there's probably something wrong in how I'm constructing the
query? And does returning this many results affect query time
substantially?

Best,
Karan

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/8bf474c2-7559-4fe4-b925-a62aedf3f09b%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(Alexander Reelsen) #2

Hey,

answering this question is sort of philosophical :slight_smile:

First, noone knows the requirements of your application. There might be
applications where it is inacceptable to not return any hits, so people
like to rather return 100k hits instead of none. Most of the time this is
not the case though.

Second, those 100k hits might be valid. Maybe you just used a generic
search term. Hard to tell from the outside, if your search quality is good
or bad. Judging from your comment, it sounds different however.

So, what can be done about it? You should try and use the analyze API,
which helps you, how the content of a field is acutally broken down and
stored in the index. This automatically helps you to understand, why a
certain term might match on this document. Also the Explain API may be used
for this. See

http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/indices-analyze.html
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-explain.html

If you want to use a nice graphical frontend for the analyze API, try the
inquisitor plugin.

In your setup another strategy might be to try another search first, and
only if that one does not match, you could use the (edge)ngram one, which
is most likely to return more hits. Also, I guess there was a reason, why
you chose that configuration in the first place. With a couple of examples,
we might help to explain, why certain documents are actually being
returned. A reproducible gist for others might help here.

Hope this helps as a start to dive deeper into your issue.

--Alex

On Sat, Jan 4, 2014 at 6:39 AM, Karan Verma karan@healthtap.com wrote:

Hi

I implemented a custom filter which uses the EdgeGram tokenizer. The
problem I face is that whether I search for something relevant or total
garbage I get a large number of hits. I suspect that this is due that fact
that I'm using an EdgeNgram tokenizer. How can I ensure the quality of the
search results and have ES return no matches for low quality results?

Does the fact that elastic search returns a large number of hits
(~100,000) mean that there's probably something wrong in how I'm
constructing the query? And does returning this many results affect query
time substantially?

Best,
Karan

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/8bf474c2-7559-4fe4-b925-a62aedf3f09b%40googlegroups.com
.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAGCwEM803SPZOcUTtd9VzAV04fwmA1%3D6W6hdKbA3iBEXEc7T%2BQ%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.


(system) #3