# Fuzzy matching and direct hit ranking **URL:** \ **Category:** Elasticsearch\ **Created:** [May 15, 2012, 6:23pm UTC](https://discuss.elastic.co/t/fuzzy-matching-and-direct-hit-ranking/7709 "2012-05-15T18:23:00Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![Woody\_Peterson](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/woody_peterson/32/830_2.png) [@Woody\_Peterson](https://discuss.elastic.co/u/Woody_Peterson)\ **Post date:** [May 15, 2012, 6:23pm UTC](https://discuss.elastic.co/t/fuzzy-matching-and-direct-hit-ranking/7709/1 "2012-05-15T18:23:00Z")
I'm trying to implement fuzzy matching, and wanted to get a sanity check, as I'm hitting what was initially a surprising use case. Let's say I have an index with documents: [{'name' =\> 'Coleman'}, {'name' =\> 'Boleman'}]. Doing a fuzzy\_like\_this search for 'coleman' will non-deterministically return either document first, whereas before I read up on the details of fuzzy search I would have expected the results to have taken distance into account. After reading some of the documentation and relevant posts on this forum, I understand that what it's doing is expanding the search to all terms in the index within a percentage-wise distance of the word. So in the above example, my current understanding is that a search for 'coleman' is literally the same thing as searching for 'coleman' and 'boleman'. First of all, is this correct? Second, is how do I achieve my desired behavior? My first thought is to do a dis\_max query with both a text and fuzzy\_like\_this. Would anyone pursue a different strategy instead? Thanks! -Woody ---
**Author:** ![rpsandiford](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rpsandiford/32/2206_2.png) [@rpsandiford](https://discuss.elastic.co/u/rpsandiford)\ **Post date:** [May 15, 2012, 8:05pm UTC](https://discuss.elastic.co/t/fuzzy-matching-and-direct-hit-ranking/7709/2 "2012-05-15T20:05:40Z")
You might want to check that you are doing a lower case at indexing time. Because - the distance between "Coleman" and "coleman" is the same as the distance between "Boleman" and "coleman" - i.e. both require a one-character replacement. Bob Sandiford | Lead Software Engineer | SirsiDynix P: 800.288.8020 X6943 | [Bob.Sandiford@sirsidynix.com](mailto:Bob.Sandiford@sirsidynix.com)[mailto:Bob.Sandiford@sirsidynix.com](mailto:Bob.Sandiford@sirsidynix.com) [www.sirsidynix.com](http://www.sirsidynix.com)[http://www.sirsidynix.com/](http://www.sirsidynix.com/) Join the conversation: Like us on Facebook! [http://www.facebook.com/SirsiDynix](http://www.facebook.com/SirsiDynix) Follow us on Twitter! [http://twitter.com/SirsiDynix](http://twitter.com/SirsiDynix) From: Woody Peterson [via ElasticSearch Users] [[mailto:ml-node+s115913n3988602h9@n3.nabble.com](mailto:ml-node+s115913n3988602h9@n3.nabble.com)] Sent: Tuesday, May 15, 2012 2:23 PM To: Bob Sandiford Subject: fuzzy matching and direct hit ranking I'm trying to implement fuzzy matching, and wanted to get a sanity check, as I'm hitting what was initially a surprising use case. Let's say I have an index with documents: [{'name' =\> 'Coleman'}, {'name' =\> 'Boleman'}]. Doing a fuzzy\_like\_this search for 'coleman' will non-deterministically return either document first, whereas before I read up on the details of fuzzy search I would have expected the results to have taken distance into account. After reading some of the documentation and relevant posts on this forum, I understand that what it's doing is expanding the search to all terms in the index within a percentage-wise distance of the word. So in the above example, my current understanding is that a search for 'coleman' is literally the same thing as searching for 'coleman' and 'boleman'. First of all, is this correct? Second, is how do I achieve my desired behavior? My first thought is to do a dis\_max query with both a text and fuzzy\_like\_this. Would anyone pursue a different strategy instead? Thanks! -Woody * * * If you reply to this email, your message will be added to the discussion below: [http://elasticsearch-users.115913.n3.nabble.com/fuzzy-matching-and-direct-hit-ranking-tp3988602.html](http://elasticsearch-users.115913.n3.nabble.com/fuzzy-matching-and-direct-hit-ranking-tp3988602.html) To start a new topic under ElasticSearch Users, email [ml-node+s115913n115913h50@n3.nabble.com](mailto:ml-node+s115913n115913h50@n3.nabble.com)[mailto:ml-node+s115913n115913h50@n3.nabble.com](mailto:ml-node+s115913n115913h50@n3.nabble.com) To unsubscribe from ElasticSearch Users, click here[http://elasticsearch-users.115913.n3.nabble.com/template/NamlServlet.jtp?macro=unsubscribe\_by\_code&node=115913&code=Ym9iLnNhbmRpZm9yZEBzaXJzaWR5bml4LmNvbXwxMTU5MTN8LTIxMTYxMTI0NTQ=](http://elasticsearch-users.115913.n3.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=115913&code=Ym9iLnNhbmRpZm9yZEBzaXJzaWR5bml4LmNvbXwxMTU5MTN8LTIxMTYxMTI0NTQ=). NAML[http://elasticsearch-users.115913.n3.nabble.com/template/NamlServlet.jtp?macro=macro\_viewer&id=instant\_html!nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify\_subscribers!nabble%3Aemail.naml-instant\_emails!nabble%3Aemail.naml-send\_instant\_email!nabble%3Aemail.naml](http://elasticsearch-users.115913.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml) ---
**Author:** ![Bob\_Sandiford](https://avatars.discourse-cdn.com/v4/letter/b/dbc845/32.png) [@Bob\_Sandiford](https://discuss.elastic.co/u/Bob_Sandiford)\ **Post date:** [May 16, 2012, 2:58pm UTC](https://discuss.elastic.co/t/fuzzy-matching-and-direct-hit-ranking/7709/3 "2012-05-16T14:58:29Z")
You might want to check that you are doing a lower case at indexing time. Because – the distance between “Coleman” and “coleman” is the same as the distance between “Boleman” and “coleman” – i.e. both require a one- character replacement. Bob. ---
**Author:** ![Woody\_Peterson](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/woody_peterson/32/830_2.png) [@Woody\_Peterson](https://discuss.elastic.co/u/Woody_Peterson)\ **Post date:** [May 16, 2012, 4:46pm UTC](https://discuss.elastic.co/t/fuzzy-matching-and-direct-hit-ranking/7709/4 "2012-05-16T16:46:25Z")
> the distance between “Coleman” and “coleman” is the same as the distance > between “Boleman” and “coleman” I tried a ton of stuff yesterday, and that might explain differences I was seeing between fuzzy\_like\_this vs 'text' searches w/ a fuziness ('text' searches go through normal tokenization, while fuzzy\_like\_this apparently doesn't). I think if your field uses the standard analyzer, it should be lowercased on index, so you would just have to lowercase it on search for fuzzy\_like\_this or use 'text', no? I eventually settled on the following, although it still exhibits this edge case bug: curl -XPUT '[http://localhost:9200/test/user/1](http://localhost:9200/test/user/1)' -d '{"name":"Coleman"}' curl -XPUT '[http://localhost:9200/test/user/2](http://localhost:9200/test/user/2)' -d '{"name":"Boleman"}' Now do the following, replacing 'Coleman' with 'Boleman', 'coleman', and 'boleman': curl -X GET "[http://localhost:9200/test/\_search?pretty=true](http://localhost:9200/test/_search?pretty=true)" -d '{"query":{"text":{"\_all":{"query":"Coleman","fuzziness":0.8}}},"explain":true}' I get a score of 0.30685282 for both documents in all search variations. The order is _not_ indeterminate as I stated in my first email, this is only seen in some cases of my particular dataset because every once in a while it will hit a document with some tiny additional factor, such as a slightly different fieldNorm, that will boost one doc over the other and appear to the user as nonsensical. This test case, however, is consistent, although it isn't clear to me what secondary sorting is going on to keep 'Coleman' always ahead of 'Boleman'; I'm assuming insertion order. -Woody On Wednesday, May 16, 2012 7:58:29 AM UTC-7, Bob Sandiford wrote: > You might want to check that you are doing a lower case at indexing > time. > > Because – the distance between “Coleman” and “coleman” is the same as > the distance between “Boleman” and “coleman” – i.e. both require a one- > character replacement. > > Bob. ---
**Author:** ![Woody\_Peterson](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/woody_peterson/32/830_2.png) [@Woody\_Peterson](https://discuss.elastic.co/u/Woody_Peterson)\ **Post date:** [May 16, 2012, 4:52pm UTC](https://discuss.elastic.co/t/fuzzy-matching-and-direct-hit-ranking/7709/5 "2012-05-16T16:52:13Z")
> curl -XPUT '[http://localhost:9200/test/user/1](http://localhost:9200/test/user/1)' -d '{"name":"Coleman"}' > curl -XPUT '[http://localhost:9200/test/user/2](http://localhost:9200/test/user/2)' -d '{"name":"Boleman"}' > ... I get a score of 0.30685282 for both documents in all search > variations I also meant to point out that I get exactly the same results (same score and everything) when doing '{"name":"coleman"}' and '{"name":"boleman"}', also. Which I would predict, but is nice to verify. On Wednesday, May 16, 2012 9:46:25 AM UTC-7, Woody Peterson wrote: > > the distance between “Coleman” and “coleman” is the same as the distance > > between “Boleman” and “coleman” > > I tried a ton of stuff yesterday, and that might explain differences I was > seeing between fuzzy\_like\_this vs 'text' searches w/ a fuziness ('text' > searches go through normal tokenization, while fuzzy\_like\_this apparently > doesn't). I think if your field uses the standard analyzer, it should be > lowercased on index, so you would just have to lowercase it on search for > fuzzy\_like\_this or use 'text', no? > > I eventually settled on the following, although it still exhibits this > edge case bug: > > curl -XPUT '[http://localhost:9200/test/user/1](http://localhost:9200/test/user/1)' -d '{"name":"Coleman"}' > curl -XPUT '[http://localhost:9200/test/user/2](http://localhost:9200/test/user/2)' -d '{"name":"Boleman"}' > > Now do the following, replacing 'Coleman' with 'Boleman', 'coleman', and > 'boleman': > > curl -X GET "[http://localhost:9200/test/\_search?pretty=true](http://localhost:9200/test/_search?pretty=true)" -d > '{"query":{"text":{"\_all":{"query":"Coleman","fuzziness":0.8}}},"explain":true}' > > I get a score of 0.30685282 for both documents in all search variations. > The order is _not_ indeterminate as I stated in my first email, this is > only seen in some cases of my particular dataset because every once in a > while it will hit a document with some tiny additional factor, such as a > slightly different fieldNorm, that will boost one doc over the other and > appear to the user as nonsensical. This test case, however, is consistent, > although it isn't clear to me what secondary sorting is going on to keep > 'Coleman' always ahead of 'Boleman'; I'm assuming insertion order. > > -Woody > > On Wednesday, May 16, 2012 7:58:29 AM UTC-7, Bob Sandiford wrote: > > > You might want to check that you are doing a lower case at indexing > > time. > > > > Because – the distance between “Coleman” and “coleman” is the same as > > the distance between “Boleman” and “coleman” – i.e. both require a one- > > character replacement. > > > > Bob. ---
**Author:** ![Woody\_Peterson](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/woody_peterson/32/830_2.png) [@Woody\_Peterson](https://discuss.elastic.co/u/Woody_Peterson)\ **Post date:** [May 17, 2012, 6:02pm UTC](https://discuss.elastic.co/t/fuzzy-matching-and-direct-hit-ranking/7709/6 "2012-05-17T18:02:55Z")
Based on Bob's reaction, I would say at least one other person would expect elasticsearch to rank direct hits above close fuzzy hits. If this is not the case, is it a bug, or am I doing it wrong? On Wednesday, May 16, 2012 9:52:13 AM UTC-7, Woody Peterson wrote: > > curl -XPUT '[http://localhost:9200/test/user/1](http://localhost:9200/test/user/1)' -d '{"name":"Coleman"}' > > curl -XPUT '[http://localhost:9200/test/user/2](http://localhost:9200/test/user/2)' -d '{"name":"Boleman"}' > > ... I get a score of 0.30685282 for both documents in all search > > variations > > I also meant to point out that I get exactly the same results (same score > and everything) when doing '{"name":"coleman"}' and '{"name":"boleman"}', > also. Which I would predict, but is nice to verify. > > On Wednesday, May 16, 2012 9:46:25 AM UTC-7, Woody Peterson wrote: > > > > the distance between “Coleman” and “coleman” is the same as the > > > distance between “Boleman” and “coleman” > > > > I tried a ton of stuff yesterday, and that might explain differences I > > was seeing between fuzzy\_like\_this vs 'text' searches w/ a fuziness ('text' > > searches go through normal tokenization, while fuzzy\_like\_this apparently > > doesn't). I think if your field uses the standard analyzer, it should be > > lowercased on index, so you would just have to lowercase it on search for > > fuzzy\_like\_this or use 'text', no? > > > > I eventually settled on the following, although it still exhibits this > > edge case bug: > > > > curl -XPUT '[http://localhost:9200/test/user/1](http://localhost:9200/test/user/1)' -d '{"name":"Coleman"}' > > curl -XPUT '[http://localhost:9200/test/user/2](http://localhost:9200/test/user/2)' -d '{"name":"Boleman"}' > > > > Now do the following, replacing 'Coleman' with 'Boleman', 'coleman', and > > 'boleman': > > > > curl -X GET "[http://localhost:9200/test/\_search?pretty=true](http://localhost:9200/test/_search?pretty=true)" -d > > '{"query":{"text":{"\_all":{"query":"Coleman","fuzziness":0.8}}},"explain":true}' > > > > I get a score of 0.30685282 for both documents in all search variations. > > The order is _not_ indeterminate as I stated in my first email, this is > > only seen in some cases of my particular dataset because every once in a > > while it will hit a document with some tiny additional factor, such as a > > slightly different fieldNorm, that will boost one doc over the other and > > appear to the user as nonsensical. This test case, however, is consistent, > > although it isn't clear to me what secondary sorting is going on to keep > > 'Coleman' always ahead of 'Boleman'; I'm assuming insertion order. > > > > -Woody > > > > On Wednesday, May 16, 2012 7:58:29 AM UTC-7, Bob Sandiford wrote: > > > > > You might want to check that you are doing a lower case at indexing > > > time. > > > > > > Because – the distance between “Coleman” and “coleman” is the same as > > > the distance between “Boleman” and “coleman” – i.e. both require a one- > > > character replacement. > > > > > > Bob. ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [May 20, 2012, 8:05pm UTC](https://discuss.elastic.co/t/fuzzy-matching-and-direct-hit-ranking/7709/7 "2012-05-20T20:05:04Z")
Yes, you will need to do text query and fuzzy and rank the text one higher... On Thu, May 17, 2012 at 8:02 PM, Woody Peterson [woody.peterson@gmail.com](mailto:woody.peterson@gmail.com)wrote: > Based on Bob's reaction, I would say at least one other person would > expect elasticsearch to rank direct hits above close fuzzy hits. If this is > not the case, is it a bug, or am I doing it wrong? > > On Wednesday, May 16, 2012 9:52:13 AM UTC-7, Woody Peterson wrote: > > > > curl -XPUT '[http://localhost:9200/test/\*\*user/1](http://localhost:9200/test/**user/1)[http://localhost:9200/test/user/1](http://localhost:9200/test/user/1)' > > > -d '{"name":"Coleman"}' > > > curl -XPUT '[http://localhost:9200/test/\*\*user/2](http://localhost:9200/test/**user/2)[http://localhost:9200/test/user/2](http://localhost:9200/test/user/2)' > > > -d '{"name":"Boleman"}' > > > ... I get a score of 0.30685282 for both documents in all search > > > variations > > > > I also meant to point out that I get exactly the same results (same score > > and everything) when doing '{"name":"coleman"}' and '{"name":"boleman"}', > > also. Which I would predict, but is nice to verify. > > > > On Wednesday, May 16, 2012 9:46:25 AM UTC-7, Woody Peterson wrote: > > > > > > the distance between “Coleman” and “coleman” is the same as the > > > > distance between “Boleman” and “coleman” > > > > > > I tried a ton of stuff yesterday, and that might explain differences I > > > was seeing between fuzzy\_like\_this vs 'text' searches w/ a fuziness ('text' > > > searches go through normal tokenization, while fuzzy\_like\_this apparently > > > doesn't). I think if your field uses the standard analyzer, it should be > > > lowercased on index, so you would just have to lowercase it on search for > > > fuzzy\_like\_this or use 'text', no? > > > > > > I eventually settled on the following, although it still exhibits this > > > edge case bug: > > > > > > curl -XPUT '[http://localhost:9200/test/\*\*user/1](http://localhost:9200/test/**user/1)[http://localhost:9200/test/user/1](http://localhost:9200/test/user/1)' > > > -d '{"name":"Coleman"}' > > > curl -XPUT '[http://localhost:9200/test/\*\*user/2](http://localhost:9200/test/**user/2)[http://localhost:9200/test/user/2](http://localhost:9200/test/user/2)' > > > -d '{"name":"Boleman"}' > > > > > > Now do the following, replacing 'Coleman' with 'Boleman', 'coleman', and > > > 'boleman': > > > > > > curl -X GET "[http://localhost:9200/test/\_\*\*search?pretty=true](http://localhost:9200/test/_**search?pretty=true)[http://localhost:9200/test/\_search?pretty=true](http://localhost:9200/test/_search?pretty=true)" > > > -d '{"query":{"text":{"\_all":{" **query":"Coleman","fuzziness":** > > > 0.8}}},"explain":true}' > > > > > > I get a score of 0.30685282 for both documents in all search variations. > > > The order is _not_ indeterminate as I stated in my first email, this is > > > only seen in some cases of my particular dataset because every once in a > > > while it will hit a document with some tiny additional factor, such as a > > > slightly different fieldNorm, that will boost one doc over the other and > > > appear to the user as nonsensical. This test case, however, is consistent, > > > although it isn't clear to me what secondary sorting is going on to keep > > > 'Coleman' always ahead of 'Boleman'; I'm assuming insertion order. > > > > > > -Woody > > > > > > On Wednesday, May 16, 2012 7:58:29 AM UTC-7, Bob Sandiford wrote: > > > > > > > You might want to check that you are doing a lower case at indexing > > > > time. > > > > > > > > Because – the distance between “Coleman” and “coleman” is the same as > > > > the distance between “Boleman” and “coleman” – i.e. both require a one- > > > > character replacement. > > > > > > > > Bob. ---
**Author:** ![Woody\_Peterson](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/woody_peterson/32/830_2.png) [@Woody\_Peterson](https://discuss.elastic.co/u/Woody_Peterson)\ **Post date:** [May 21, 2012, 4:30pm UTC](https://discuss.elastic.co/t/fuzzy-matching-and-direct-hit-ranking/7709/8 "2012-05-21T16:30:37Z")
This thread [Carbon60: Managed Cloud Services](http://www.gossamer-threads.com/lists/lucene/java-user/144162) has the following comment that suggests it is a lucene option which strategy to take: > it should score "farming" higher than "farmin" by > default, but the default rewrite mode also takes TF/IDF into account (in > addition). You can change that by a different rewrite method: > > The default is: [MultiTermQuery.TopTermsScoringBooleanQueryRewrite (Lucene 3.5.0 API)](http://goo.gl/JhHOA) (which combines the standard vector > model with additionally boosting exact matches - we have that for > backwards > compatibility only, its not what most users expect) > > The better one is: [MultiTermQuery.TopTermsBoostOnlyBooleanQueryRewrite (Lucene 3.5.0 API)](http://goo.gl/0eJ47), which does not take TF/IDF into > account and only boosts by levensthein distance. > > You can disable fuzzy boosting altogether: > Additionally [ScoringRewrite (Lucene 3.5.0 API)](http://goo.gl/VWlkW) provides two other scoring models > (TF/IDF > only, no boosting - or constant score at all) Would it make sense to expose these options for use in elasticsearch? On Sunday, May 20, 2012 1:05:04 PM UTC-7, kimchy wrote: > Yes, you will need to do text query and fuzzy and rank the text one > higher... > > On Thu, May 17, 2012 at 8:02 PM, Woody Peterson wrote: > > > Based on Bob's reaction, I would say at least one other person would > > expect elasticsearch to rank direct hits above close fuzzy hits. If this is > > not the case, is it a bug, or am I doing it wrong? > > > > On Wednesday, May 16, 2012 9:52:13 AM UTC-7, Woody Peterson wrote: > > > > > > curl -XPUT '[http://localhost:9200/test/\*\*user/1](http://localhost:9200/test/**user/1)[http://localhost:9200/test/user/1](http://localhost:9200/test/user/1)' > > > > -d '{"name":"Coleman"}' > > > > curl -XPUT '[http://localhost:9200/test/\*\*user/2](http://localhost:9200/test/**user/2)[http://localhost:9200/test/user/2](http://localhost:9200/test/user/2)' > > > > -d '{"name":"Boleman"}' > > > > ... I get a score of 0.30685282 for both documents in all search > > > > variations > > > > > > I also meant to point out that I get exactly the same results (same > > > score and everything) when doing '{"name":"coleman"}' and > > > '{"name":"boleman"}', also. Which I would predict, but is nice to verify. > > > > > > On Wednesday, May 16, 2012 9:46:25 AM UTC-7, Woody Peterson wrote: > > > > > > > > the distance between “Coleman” and “coleman” is the same as the > > > > > distance between “Boleman” and “coleman” > > > > > > > > I tried a ton of stuff yesterday, and that might explain differences I > > > > was seeing between fuzzy\_like\_this vs 'text' searches w/ a fuziness ('text' > > > > searches go through normal tokenization, while fuzzy\_like\_this apparently > > > > doesn't). I think if your field uses the standard analyzer, it should be > > > > lowercased on index, so you would just have to lowercase it on search for > > > > fuzzy\_like\_this or use 'text', no? > > > > > > > > I eventually settled on the following, although it still exhibits this > > > > edge case bug: > > > > > > > > curl -XPUT '[http://localhost:9200/test/\*\*user/1](http://localhost:9200/test/**user/1)[http://localhost:9200/test/user/1](http://localhost:9200/test/user/1)' > > > > -d '{"name":"Coleman"}' > > > > curl -XPUT '[http://localhost:9200/test/\*\*user/2](http://localhost:9200/test/**user/2)[http://localhost:9200/test/user/2](http://localhost:9200/test/user/2)' > > > > -d '{"name":"Boleman"}' > > > > > > > > Now do the following, replacing 'Coleman' with 'Boleman', 'coleman', > > > > and 'boleman': > > > > > > > > curl -X GET "[http://localhost:9200/test/\_\*\*search?pretty=true](http://localhost:9200/test/_**search?pretty=true)[http://localhost:9200/test/\_search?pretty=true](http://localhost:9200/test/_search?pretty=true)" > > > > -d '{"query":{"text":{"\_all":{" **query":"Coleman","fuzziness":** > > > > 0.8}}},"explain":true}' > > > > > > > > I get a score of 0.30685282 for both documents in all search > > > > variations. The order is _not_ indeterminate as I stated in my first email, > > > > this is only seen in some cases of my particular dataset because every once > > > > in a while it will hit a document with some tiny additional factor, such as > > > > a slightly different fieldNorm, that will boost one doc over the other and > > > > appear to the user as nonsensical. This test case, however, is consistent, > > > > although it isn't clear to me what secondary sorting is going on to keep > > > > 'Coleman' always ahead of 'Boleman'; I'm assuming insertion order. > > > > > > > > -Woody > > > > > > > > On Wednesday, May 16, 2012 7:58:29 AM UTC-7, Bob Sandiford wrote: > > > > > > > > > You might want to check that you are doing a lower case at indexing > > > > > time. > > > > > > > > > > Because – the distance between “Coleman” and “coleman” is the same as > > > > > the distance between “Boleman” and “coleman” – i.e. both require a > > > > > one- > > > > > character replacement. > > > > > > > > > > Bob. ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [May 23, 2012, 9:57pm UTC](https://discuss.elastic.co/t/fuzzy-matching-and-direct-hit-ranking/7709/9 "2012-05-23T21:57:43Z")
Sadly, the Lucene query parser does not expose the ability to set the rewrite method for fuzzy query, but we can work around that. You can set the rewrite method for the query\_string, but it applies to wildcard and prefix queries. We can add a fuzzy rewrite method option as well here... On Mon, May 21, 2012 at 6:30 PM, Woody Peterson [woody.peterson@gmail.com](mailto:woody.peterson@gmail.com)wrote: > This thread [Carbon60: Managed Cloud Services](http://www.gossamer-threads.com/lists/lucene/java-user/144162has) the following comment that suggests it is a lucene option which > strategy to take: > > > it should score "farming" higher than "farmin" by > > default, but the default rewrite mode also takes TF/IDF into account (in > > addition). You can change that by a different rewrite method: > > > > The default is: [MultiTermQuery.TopTermsScoringBooleanQueryRewrite (Lucene 3.5.0 API)](http://goo.gl/JhHOA) (which combines the standard vector > > model with additionally boosting exact matches - we have that for > > backwards > > compatibility only, its not what most users expect) > > > > The better one is: [MultiTermQuery.TopTermsBoostOnlyBooleanQueryRewrite (Lucene 3.5.0 API)](http://goo.gl/0eJ47), which does not take TF/IDF into > > account and only boosts by levensthein distance. > > > > You can disable fuzzy boosting altogether: > > Additionally [ScoringRewrite (Lucene 3.5.0 API)](http://goo.gl/VWlkW) provides two other scoring models > > (TF/IDF > > only, no boosting - or constant score at all) > > Would it make sense to expose these options for use in elasticsearch? > > On Sunday, May 20, 2012 1:05:04 PM UTC-7, kimchy wrote: > > > Yes, you will need to do text query and fuzzy and rank the text one > > higher... > > > > On Thu, May 17, 2012 at 8:02 PM, Woody Peterson wrote: > > > > Based on Bob's reaction, I would say at least one other person would > > > > > expect elasticsearch to rank direct hits above close fuzzy hits. If this is > > > not the case, is it a bug, or am I doing it wrong? > > > > > > On Wednesday, May 16, 2012 9:52:13 AM UTC-7, Woody Peterson wrote: > > > > > > > > curl -XPUT '[http://localhost:9200/test/\*\*us\*\*er/1](http://localhost:9200/test/ **us** er/1)[http://localhost:9200/test/user/1](http://localhost:9200/test/user/1)' > > > > > -d '{"name":"Coleman"}' > > > > > curl -XPUT '[http://localhost:9200/test/\*\*us\*\*er/2](http://localhost:9200/test/ **us** er/2)[http://localhost:9200/test/user/2](http://localhost:9200/test/user/2)' > > > > > -d '{"name":"Boleman"}' > > > > > ... I get a score of 0.30685282 for both documents in all search > > > > > variations > > > > > > > > I also meant to point out that I get exactly the same results (same > > > > score and everything) when doing '{"name":"coleman"}' and > > > > '{"name":"boleman"}', also. Which I would predict, but is nice to verify. > > > > > > > > On Wednesday, May 16, 2012 9:46:25 AM UTC-7, Woody Peterson wrote: > > > > > > > > > > the distance between “Coleman” and “coleman” is the same as the > > > > > > distance between “Boleman” and “coleman” > > > > > > > > > > I tried a ton of stuff yesterday, and that might explain differences I > > > > > was seeing between fuzzy\_like\_this vs 'text' searches w/ a fuziness ('text' > > > > > searches go through normal tokenization, while fuzzy\_like\_this apparently > > > > > doesn't). I think if your field uses the standard analyzer, it should be > > > > > lowercased on index, so you would just have to lowercase it on search for > > > > > fuzzy\_like\_this or use 'text', no? > > > > > > > > > > I eventually settled on the following, although it still exhibits this > > > > > edge case bug: > > > > > > > > > > curl -XPUT '[http://localhost:9200/test/\*\*us\*\*er/1](http://localhost:9200/test/ **us** er/1)[http://localhost:9200/test/user/1](http://localhost:9200/test/user/1)' > > > > > -d '{"name":"Coleman"}' > > > > > curl -XPUT '[http://localhost:9200/test/\*\*us\*\*er/2](http://localhost:9200/test/ **us** er/2)[http://localhost:9200/test/user/2](http://localhost:9200/test/user/2)' > > > > > -d '{"name":"Boleman"}' > > > > > > > > > > Now do the following, replacing 'Coleman' with 'Boleman', 'coleman', > > > > > and 'boleman': > > > > > > > > > > curl -X GET "[http://localhost:9200/test/\_\*\*s\*\*earch?pretty=true](http://localhost:9200/test/_ **s** earch?pretty=true)[http://localhost:9200/test/\_search?pretty=true](http://localhost:9200/test/_search?pretty=true)" > > > > > -d '{"query":{"text":{"\_all":{" **qu** ery":"Coleman","fuzziness": **0.** > > > > > 8}}},"explain":true}' > > > > > > > > > > I get a score of 0.30685282 for both documents in all search > > > > > variations. The order is _not_ indeterminate as I stated in my first email, > > > > > this is only seen in some cases of my particular dataset because every once > > > > > in a while it will hit a document with some tiny additional factor, such as > > > > > a slightly different fieldNorm, that will boost one doc over the other and > > > > > appear to the user as nonsensical. This test case, however, is consistent, > > > > > although it isn't clear to me what secondary sorting is going on to keep > > > > > 'Coleman' always ahead of 'Boleman'; I'm assuming insertion order. > > > > > > > > > > -Woody > > > > > > > > > > On Wednesday, May 16, 2012 7:58:29 AM UTC-7, Bob Sandiford wrote: > > > > > > > > > > > You might want to check that you are doing a lower case at indexing > > > > > > time. > > > > > > > > > > > > Because – the distance between “Coleman” and “coleman” is the same as > > > > > > the distance between “Boleman” and “coleman” – i.e. both require a > > > > > > one- > > > > > > character replacement. > > > > > > > > > > > > Bob. ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [May 23, 2012, 10:28pm UTC](https://discuss.elastic.co/t/fuzzy-matching-and-direct-hit-ranking/7709/10 "2012-05-23T22:28:45Z")
Had another look, and it seems like we can hook specific fuzzy settings to the Lucene query parser and in other places, opened an issue: [Query DSL: Add more fuzzy options in different queries (text, query\_string/field) · Issue #1974 · elastic/elasticsearch · GitHub](https://github.com/elasticsearch/elasticsearch/issues/1974). On Wed, May 23, 2012 at 11:57 PM, Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com) wrote: > Sadly, the Lucene query parser does not expose the ability to set the > rewrite method for fuzzy query, but we can work around that. You can set > the rewrite method for the query\_string, but it applies to wildcard and > prefix queries. We can add a fuzzy rewrite method option as well here... > > On Mon, May 21, 2012 at 6:30 PM, Woody Peterson [woody.peterson@gmail.com](mailto:woody.peterson@gmail.com)wrote: > > > This thread [Carbon60: Managed Cloud Services](http://www.gossamer-threads.com/lists/lucene/java-user/144162has) the following comment that suggests it is a lucene option which > > strategy to take: > > > > > it should score "farming" higher than "farmin" by > > > default, but the default rewrite mode also takes TF/IDF into account > > > (in > > > addition). You can change that by a different rewrite method: > > > > > > The default is: [MultiTermQuery.TopTermsScoringBooleanQueryRewrite (Lucene 3.5.0 API)](http://goo.gl/JhHOA) (which combines the standard > > > vector > > > model with additionally boosting exact matches - we have that for > > > backwards > > > compatibility only, its not what most users expect) > > > > > > The better one is: [MultiTermQuery.TopTermsBoostOnlyBooleanQueryRewrite (Lucene 3.5.0 API)](http://goo.gl/0eJ47), which does not take TF/IDF > > > into > > > account and only boosts by levensthein distance. > > > > > > You can disable fuzzy boosting altogether: > > > Additionally [ScoringRewrite (Lucene 3.5.0 API)](http://goo.gl/VWlkW) provides two other scoring models > > > (TF/IDF > > > only, no boosting - or constant score at all) > > > > Would it make sense to expose these options for use in elasticsearch? > > > > On Sunday, May 20, 2012 1:05:04 PM UTC-7, kimchy wrote: > > > > > Yes, you will need to do text query and fuzzy and rank the text one > > > higher... > > > > > > On Thu, May 17, 2012 at 8:02 PM, Woody Peterson wrote: > > > > > > Based on Bob's reaction, I would say at least one other person would > > > > > > > expect elasticsearch to rank direct hits above close fuzzy hits. If this is > > > > not the case, is it a bug, or am I doing it wrong? > > > > > > > > On Wednesday, May 16, 2012 9:52:13 AM UTC-7, Woody Peterson wrote: > > > > > > > > > > curl -XPUT '[http://localhost:9200/test/\*\*us\*\*er/1](http://localhost:9200/test/ **us** er/1)[http://localhost:9200/test/user/1](http://localhost:9200/test/user/1)' > > > > > > -d '{"name":"Coleman"}' > > > > > > curl -XPUT '[http://localhost:9200/test/\*\*us\*\*er/2](http://localhost:9200/test/ **us** er/2)[http://localhost:9200/test/user/2](http://localhost:9200/test/user/2)' > > > > > > -d '{"name":"Boleman"}' > > > > > > ... I get a score of 0.30685282 for both documents in all search > > > > > > variations > > > > > > > > > > I also meant to point out that I get exactly the same results (same > > > > > score and everything) when doing '{"name":"coleman"}' and > > > > > '{"name":"boleman"}', also. Which I would predict, but is nice to verify. > > > > > > > > > > On Wednesday, May 16, 2012 9:46:25 AM UTC-7, Woody Peterson wrote: > > > > > > > > > > > > the distance between “Coleman” and “coleman” is the same as the > > > > > > > distance between “Boleman” and “coleman” > > > > > > > > > > > > I tried a ton of stuff yesterday, and that might explain differences > > > > > > I was seeing between fuzzy\_like\_this vs 'text' searches w/ a fuziness > > > > > > ('text' searches go through normal tokenization, while fuzzy\_like\_this > > > > > > apparently doesn't). I think if your field uses the standard analyzer, it > > > > > > should be lowercased on index, so you would just have to lowercase it on > > > > > > search for fuzzy\_like\_this or use 'text', no? > > > > > > > > > > > > I eventually settled on the following, although it still exhibits > > > > > > this edge case bug: > > > > > > > > > > > > curl -XPUT '[http://localhost:9200/test/\*\*us\*\*er/1](http://localhost:9200/test/ **us** er/1)[http://localhost:9200/test/user/1](http://localhost:9200/test/user/1)' > > > > > > -d '{"name":"Coleman"}' > > > > > > curl -XPUT '[http://localhost:9200/test/\*\*us\*\*er/2](http://localhost:9200/test/ **us** er/2)[http://localhost:9200/test/user/2](http://localhost:9200/test/user/2)' > > > > > > -d '{"name":"Boleman"}' > > > > > > > > > > > > Now do the following, replacing 'Coleman' with 'Boleman', 'coleman', > > > > > > and 'boleman': > > > > > > > > > > > > curl -X GET "[http://localhost:9200/test/\_\*\*s\*\*earch?pretty=true](http://localhost:9200/test/_ **s** earch?pretty=true)[http://localhost:9200/test/\_search?pretty=true](http://localhost:9200/test/_search?pretty=true)" > > > > > > -d '{"query":{"text":{"\_all":{" **qu** ery":"Coleman","fuzziness":\*_0._ > > > > > > \*8}}},"explain":true}' > > > > > > > > > > > > I get a score of 0.30685282 for both documents in all search > > > > > > variations. The order is _not_ indeterminate as I stated in my first email, > > > > > > this is only seen in some cases of my particular dataset because every once > > > > > > in a while it will hit a document with some tiny additional factor, such as > > > > > > a slightly different fieldNorm, that will boost one doc over the other and > > > > > > appear to the user as nonsensical. This test case, however, is consistent, > > > > > > although it isn't clear to me what secondary sorting is going on to keep > > > > > > 'Coleman' always ahead of 'Boleman'; I'm assuming insertion order. > > > > > > > > > > > > -Woody > > > > > > > > > > > > On Wednesday, May 16, 2012 7:58:29 AM UTC-7, Bob Sandiford wrote: > > > > > > > > > > > > > You might want to check that you are doing a lower case at indexing > > > > > > > time. > > > > > > > > > > > > > > Because – the distance between “Coleman” and “coleman” is the same > > > > > > > as > > > > > > > the distance between “Boleman” and “coleman” – i.e. both require a > > > > > > > one- > > > > > > > character replacement. > > > > > > > > > > > > > > Bob. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 3:27am UTC](https://discuss.elastic.co/t/fuzzy-matching-and-direct-hit-ranking/7709/11 "2017-07-06T03:27:26Z")