# Can't get nGram indexing / querying to work as expected **URL:** \ **Category:** Elasticsearch\ **Created:** [May 8, 2011, 3:17pm UTC](https://discuss.elastic.co/t/cant-get-ngram-indexing-querying-to-work-as-expected/4363 "2011-05-08T15:17:57Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![hukl](https://avatars.discourse-cdn.com/v4/letter/h/ea5d25/32.png) [@hukl](https://discuss.elastic.co/u/hukl)\ **Post date:** [May 8, 2011, 3:17pm UTC](https://discuss.elastic.co/t/cant-get-ngram-indexing-querying-to-work-as-expected/4363/1 "2011-05-08T15:17:57Z")
Hey there, I have already spent half the day in the irc channel and karmi as well as clintongormley were really helpful but in the end my example still does not work as expected. What I want to achieve: I have docs which include an uri field like { uri : "[http://www.foobar.com](http://www.foobar.com)" } And I want to be able to find that doc by searching for "foo" { uri : "[http://www.mylatestwebsite.com](http://www.mylatestwebsite.com)" } And I want to be able to find it by searching for "latest" Now with the standard analyzer / tokenizer it doesn't split the domain after the dots as there is no whitespace and therefore the dots are considered as part of the token. Also its quite common for domains to be not seperated by tokens like the one above. Now collin, karmi and kimchy suggested using ngram for this and collin even provided an example which unfortunately did not work. So I produced a minimal example based on collins which I would expect to work but doesn't and I'd love to hear about any suggestions how to make this work. My ES Session for this Problem looks like this: [https://gist.github.com/gists/961418](https://gist.github.com/gists/961418) Kind regards, John ---
**Author:** ![Paul\_Loy](https://avatars.discourse-cdn.com/v4/letter/p/ad7895/32.png) [@Paul\_Loy](https://discuss.elastic.co/u/Paul_Loy)\ **Post date:** [May 8, 2011, 5:15pm UTC](https://discuss.elastic.co/t/cant-get-ngram-indexing-querying-to-work-as-expected/4363/2 "2011-05-08T17:15:18Z")
After a bit of playing I did this: $ curl -XPUT '[http://127.0.0.1:9200/test/website4/\_mapping?pretty=1](http://127.0.0.1:9200/test/website4/_mapping?pretty=1)' -d ' { "website" : { "properties" : { "uri" : { "type" : "string", "include\_in\_all" : 0, "index\_analyzer" : "ascii\_ngram", "search\_analyzer" : "ascii\_std" } } } } ' { "ok" : true, "acknowledged" : true } $ curl -XPOST '[http://127.0.0.1:9200/test/website4?pretty=1](http://127.0.0.1:9200/test/website4?pretty=1)' -d ' { "uri" : "[http://www.heise.de](http://www.heise.de)" } ' { "ok" : true, "\_index" : "test", "\_type" : "website4", "\_id" : "j3KS0Py1TWC7JjnOWeeCIg", "\_version" : 1 } $ _curl -XPOST '[http://127.0.0.1:9200/test/\_refresh](http://127.0.0.1:9200/test/_refresh)'_ {"ok":true,"\_shards":{"total":10,"successful":5,"failed":0}} $ curl -XGET '[http://127.0.0.1:9200/test/website4/\_search?pretty=1](http://127.0.0.1:9200/test/website4/_search?pretty=1)' -d ' { "query" : { "field" : { "uri" : "heis" } } } ' { "took" : 4, "timed\_out" : false, "\_shards" : { "total" : 5, "successful" : 5, "failed" : 0 }, \*"hits" : { "total" : 1, "max\_score" : 0.28602687, "hits" : [ { "\_index" : "test", "\_type" : "website4", "\_id" : "j3KS0Py1TWC7JjnOWeeCIg", "\_score" : 0.28602687, "\_source" : { "uri" : "[http://www.heise.de](http://www.heise.de)" } ``` } ] ``` }\* } Putting a refresh in there works. On Sun, May 8, 2011 at 4:17 PM, hukl [jpbader@gmail.com](mailto:jpbader@gmail.com) wrote: > Hey there, > > I have already spent half the day in the irc channel and karmi as well > as clintongormley were really helpful but in the end my example still > does not work as expected. > > What I want to achieve: > > I have docs which include an uri field like > > { uri : "[http://www.foobar.com](http://www.foobar.com)" } > > And I want to be able to find that doc by searching for "foo" > > { uri : "[http://www.mylatestwebsite.com](http://www.mylatestwebsite.com)" } > > And I want to be able to find it by searching for "latest" > > Now with the standard analyzer / tokenizer it doesn't split the domain > after the dots as there is no whitespace and therefore the dots are > considered as part of the token. Also its quite common for domains to > be not seperated by tokens like the one above. > > Now collin, karmi and kimchy suggested using ngram for this and collin > even provided an example which unfortunately did not work. So I > produced a minimal example based on collins which I would expect to > work but doesn't and I'd love to hear about any suggestions how to > make this work. > > My ES Session for this Problem looks like this: > > [https://gist.github.com/gists/961418](https://gist.github.com/gists/961418) > > Kind regards, John ## -- Paul Loy [paul@keteracel.com](mailto:paul@keteracel.com) [http://uk.linkedin.com/in/paulloy](http://uk.linkedin.com/in/paulloy) ---
**Author:** ![Paul\_Loy](https://avatars.discourse-cdn.com/v4/letter/p/ad7895/32.png) [@Paul\_Loy](https://discuss.elastic.co/u/Paul_Loy)\ **Post date:** [May 8, 2011, 5:22pm UTC](https://discuss.elastic.co/t/cant-get-ngram-indexing-querying-to-work-as-expected/4363/3 "2011-05-08T17:22:45Z")
Actually scratch that. What worked was changing the search analyser to ascii\_ngram. I think the default n for ngram is min 2, max 3. So since heis is 4 characters it will not match any of the tokens in the index. On Sun, May 8, 2011 at 6:15 PM, Paul Loy [keteracel@gmail.com](mailto:keteracel@gmail.com) wrote: > After a bit of playing I did this: > > $ curl -XPUT '[http://127.0.0.1:9200/test/website4/\_mapping?pretty=1](http://127.0.0.1:9200/test/website4/_mapping?pretty=1)' -d ' > { > "website" : { > "properties" : { > "uri" : { > "type" : "string", > "include\_in\_all" : 0, > "index\_analyzer" : "ascii\_ngram", > "search\_analyzer" : "ascii\_std" > } > } > } > } > ' > { > "ok" : true, > "acknowledged" : true > } > > $ curl -XPOST '[http://127.0.0.1:9200/test/website4?pretty=1](http://127.0.0.1:9200/test/website4?pretty=1)' -d ' > { > "uri" : "[http://www.heise.de](http://www.heise.de)" > } > ' > { > "ok" : true, > "\_index" : "test", > "\_type" : "website4", > "\_id" : "j3KS0Py1TWC7JjnOWeeCIg", > "\_version" : 1 > } > > $ _curl -XPOST '[http://127.0.0.1:9200/test/\_refresh](http://127.0.0.1:9200/test/_refresh)'_ > {"ok":true,"\_shards":{"total":10,"successful":5,"failed":0}} > > $ curl -XGET '[http://127.0.0.1:9200/test/website4/\_search?pretty=1](http://127.0.0.1:9200/test/website4/_search?pretty=1)' -d ' > { > "query" : { > "field" : { > "uri" : "heis" > } > } > } > ' > { > "took" : 4, > "timed\_out" : false, > "\_shards" : { > "total" : 5, > "successful" : 5, > "failed" : 0 > }, > \*"hits" : { > "total" : 1, > "max\_score" : 0.28602687, > "hits" : [ { > "\_index" : "test", > "\_type" : "website4", > "\_id" : "j3KS0Py1TWC7JjnOWeeCIg", > "\_score" : 0.28602687, "\_source" : > { > "uri" : "[http://www.heise.de](http://www.heise.de)" > } > > ``` > } ] > > ``` > > }\* > } > > Putting a refresh in there works. > > On Sun, May 8, 2011 at 4:17 PM, hukl [jpbader@gmail.com](mailto:jpbader@gmail.com) wrote: > > > Hey there, > > > > I have already spent half the day in the irc channel and karmi as well > > as clintongormley were really helpful but in the end my example still > > does not work as expected. > > > > What I want to achieve: > > > > I have docs which include an uri field like > > > > { uri : "[http://www.foobar.com](http://www.foobar.com)" } > > > > And I want to be able to find that doc by searching for "foo" > > > > { uri : "[http://www.mylatestwebsite.com](http://www.mylatestwebsite.com)" } > > > > And I want to be able to find it by searching for "latest" > > > > Now with the standard analyzer / tokenizer it doesn't split the domain > > after the dots as there is no whitespace and therefore the dots are > > considered as part of the token. Also its quite common for domains to > > be not seperated by tokens like the one above. > > > > Now collin, karmi and kimchy suggested using ngram for this and collin > > even provided an example which unfortunately did not work. So I > > produced a minimal example based on collins which I would expect to > > work but doesn't and I'd love to hear about any suggestions how to > > make this work. > > > > My ES Session for this Problem looks like this: > > > > [https://gist.github.com/gists/961418](https://gist.github.com/gists/961418) > > > > Kind regards, John > > ## -- > > Paul Loy > [paul@keteracel.com](mailto:paul@keteracel.com) > [Paul Loy - Amihan Entertainment | LinkedIn](http://uk.linkedin.com/in/paulloy) ## -- Paul Loy [paul@keteracel.com](mailto:paul@keteracel.com) [http://uk.linkedin.com/in/paulloy](http://uk.linkedin.com/in/paulloy) ---
**Author:** ![hukl](https://avatars.discourse-cdn.com/v4/letter/h/ea5d25/32.png) [@hukl](https://discuss.elastic.co/u/hukl)\ **Post date:** [May 8, 2011, 5:56pm UTC](https://discuss.elastic.co/t/cant-get-ngram-indexing-querying-to-work-as-expected/4363/4 "2011-05-08T17:56:26Z")
Thank you! Indeed this makes sense and returns results but the problem is that when I enter more urls it will return them all on each query. This was also mentioned by collin in this thread: [http://groups.google.com/a/elasticsearch.com/group/users/browse\_thread/thread/cd08abb8ba01b0d4/38c0ed74e263252a?lnk=gst&q=ngram#38c0ed74e263252a](http://groups.google.com/a/elasticsearch.com/group/users/browse_thread/thread/cd08abb8ba01b0d4/38c0ed74e263252a?lnk=gst&q=ngram#38c0ed74e263252a) where he suggested to use a different search tokenizer. So the question would be which search analyzer / tokenizer would fit? Or is it a bug? Kind regards, John On May 8, 7:22 pm, Paul Loy [ketera...@gmail.com](mailto:ketera...@gmail.com) wrote: > Actually scratch that. What worked was changing the search analyser to > ascii\_ngram. I think the default n for ngram is min 2, max 3. So since heis > is 4 characters it will not match any of the tokens in the index. > > On Sun, May 8, 2011 at 6:15 PM, Paul Loy [ketera...@gmail.com](mailto:ketera...@gmail.com) wrote: > > > After a bit of playing I did this: > > > $ curl -XPUT '[http://127.0.0.1:9200/test/website4/\_mapping?pretty=1'-d](http://127.0.0.1:9200/test/website4/_mapping?pretty=1'-d) ' > > { > > "website" : { > > "properties" : { > > "uri" : { > > "type" : "string", > > "include\_in\_all" : 0, > > "index\_analyzer" : "ascii\_ngram", > > "search\_analyzer" : "ascii\_std" > > } > > } > > } > > } > > ' > > { > > "ok" : true, > > "acknowledged" : true > > } > > > $ curl -XPOST '[http://127.0.0.1:9200/test/website4?pretty=1'-d](http://127.0.0.1:9200/test/website4?pretty=1'-d) ' > > { > > "uri" : "[http://www.heise.de](http://www.heise.de)" > > } > > ' > > { > > "ok" : true, > > "\_index" : "test", > > "\_type" : "website4", > > "\_id" : "j3KS0Py1TWC7JjnOWeeCIg", > > "\_version" : 1 > > } > > > $ _curl -XPOST '[http://127.0.0.1:9200/test/\_refresh](http://127.0.0.1:9200/test/_refresh)'_ > > {"ok":true,"\_shards":{"total":10,"successful":5,"failed":0}} > > > $ curl -XGET '[http://127.0.0.1:9200/test/website4/\_search?pretty=1'-d](http://127.0.0.1:9200/test/website4/_search?pretty=1'-d) ' > > { > > "query" : { > > "field" : { > > "uri" : "heis" > > } > > } > > } > > ' > > { > > "took" : 4, > > "timed\_out" : false, > > "\_shards" : { > > "total" : 5, > > "successful" : 5, > > "failed" : 0 > > }, > > \*"hits" : { > > "total" : 1, > > "max\_score" : 0.28602687, > > "hits" : [ { > > "\_index" : "test", > > "\_type" : "website4", > > "\_id" : "j3KS0Py1TWC7JjnOWeeCIg", > > "\_score" : 0.28602687, "\_source" : > > { > > "uri" : "[http://www.heise.de](http://www.heise.de)" > > } > > > ``` > > } ] > > > > ``` > > > > }\* > > } > > > Putting a refresh in there works. > > > On Sun, May 8, 2011 at 4:17 PM, hukl [jpba...@gmail.com](mailto:jpba...@gmail.com) wrote: > > > > Hey there, > > > > I have already spent half the day in the irc channel and karmi as well > > > as clintongormley were really helpful but in the end my example still > > > does not work as expected. > > > > What I want to achieve: > > > > I have docs which include an uri field like > > > > { uri : "[http://www.foobar.com](http://www.foobar.com)" } > > > > And I want to be able to find that doc by searching for "foo" > > > > { uri : "[http://www.mylatestwebsite.com](http://www.mylatestwebsite.com)" } > > > > And I want to be able to find it by searching for "latest" > > > > Now with the standard analyzer / tokenizer it doesn't split the domain > > > after the dots as there is no whitespace and therefore the dots are > > > considered as part of the token. Also its quite common for domains to > > > be not seperated by tokens like the one above. > > > > Now collin, karmi and kimchy suggested using ngram for this and collin > > > even provided an example which unfortunately did not work. So I > > > produced a minimal example based on collins which I would expect to > > > work but doesn't and I'd love to hear about any suggestions how to > > > make this work. > > > > My ES Session for this Problem looks like this: > > > > [https://gist.github.com/gists/961418](https://gist.github.com/gists/961418) > > > > Kind regards, John > > > ## -- > > > > Paul Loy > > [p...@keteracel.com](mailto:p...@keteracel.com) > > [http://uk.linkedin.com/in/paulloy](http://uk.linkedin.com/in/paulloy) > > ## -- > > Paul Loy > p...@keteracel.comhttp://uk.linkedin.com/in/paulloy ---
**Author:** ![Paul\_Loy](https://avatars.discourse-cdn.com/v4/letter/p/ad7895/32.png) [@Paul\_Loy](https://discuss.elastic.co/u/Paul_Loy)\ **Post date:** [May 8, 2011, 6:07pm UTC](https://discuss.elastic.co/t/cant-get-ngram-indexing-querying-to-work-as-expected/4363/5 "2011-05-08T18:07:32Z")
Yup, you are correct. That's a shame as I was going to use ngram for username search ☹ On Sun, May 8, 2011 at 6:56 PM, hukl [jpbader@gmail.com](mailto:jpbader@gmail.com) wrote: > Thank you! Indeed this makes sense and returns results but the problem > is that when I enter more urls it will return them all on each query. > > This was also mentioned by collin in this thread: > > [http://groups.google.com/a/elasticsearch.com/group/users/browse\_thread/thread/cd08abb8ba01b0d4/38c0ed74e263252a?lnk=gst&q=ngram#38c0ed74e263252a](http://groups.google.com/a/elasticsearch.com/group/users/browse_thread/thread/cd08abb8ba01b0d4/38c0ed74e263252a?lnk=gst&q=ngram#38c0ed74e263252a) > > where he suggested to use a different search tokenizer. > > So the question would be which search analyzer / tokenizer would fit? > Or is it a bug? > > Kind regards, John > > On May 8, 7:22 pm, Paul Loy [ketera...@gmail.com](mailto:ketera...@gmail.com) wrote: > > > Actually scratch that. What worked was changing the search analyser to > > ascii\_ngram. I think the default n for ngram is min 2, max 3. So since > > heis > > is 4 characters it will not match any of the tokens in the index. > > > > On Sun, May 8, 2011 at 6:15 PM, Paul Loy [ketera...@gmail.com](mailto:ketera...@gmail.com) wrote: > > > > > After a bit of playing I did this: > > > > > $ curl -XPUT '[http://127.0.0.1:9200/test/website4/\_mapping?pretty=1'-d'](http://127.0.0.1:9200/test/website4/_mapping?pretty=1'-d') > > > { > > > "website" : { > > > "properties" : { > > > "uri" : { > > > "type" : "string", > > > "include\_in\_all" : 0, > > > "index\_analyzer" : "ascii\_ngram", > > > "search\_analyzer" : "ascii\_std" > > > } > > > } > > > } > > > } > > > ' > > > { > > > "ok" : true, > > > "acknowledged" : true > > > } > > > > > $ curl -XPOST '[http://127.0.0.1:9200/test/website4?pretty=1'-d](http://127.0.0.1:9200/test/website4?pretty=1'-d) ' > > > { > > > "uri" : "[http://www.heise.de](http://www.heise.de)" > > > } > > > ' > > > { > > > "ok" : true, > > > "\_index" : "test", > > > "\_type" : "website4", > > > "\_id" : "j3KS0Py1TWC7JjnOWeeCIg", > > > "\_version" : 1 > > > } > > > > > $ _curl -XPOST '[http://127.0.0.1:9200/test/\_refresh](http://127.0.0.1:9200/test/_refresh)'_ > > > {"ok":true,"\_shards":{"total":10,"successful":5,"failed":0}} > > > > > $ curl -XGET '[http://127.0.0.1:9200/test/website4/\_search?pretty=1'-d'](http://127.0.0.1:9200/test/website4/_search?pretty=1'-d') > > > { > > > "query" : { > > > "field" : { > > > "uri" : "heis" > > > } > > > } > > > } > > > ' > > > { > > > "took" : 4, > > > "timed\_out" : false, > > > "\_shards" : { > > > "total" : 5, > > > "successful" : 5, > > > "failed" : 0 > > > }, > > > \*"hits" : { > > > "total" : 1, > > > "max\_score" : 0.28602687, > > > "hits" : [ { > > > "\_index" : "test", > > > "\_type" : "website4", > > > "\_id" : "j3KS0Py1TWC7JjnOWeeCIg", > > > "\_score" : 0.28602687, "\_source" : > > > { > > > "uri" : "[http://www.heise.de](http://www.heise.de)" > > > } > > > > > ``` > > > } ] > > > > > > ``` > > > > > > }\* > > > } > > > > > Putting a refresh in there works. > > > > > On Sun, May 8, 2011 at 4:17 PM, hukl [jpba...@gmail.com](mailto:jpba...@gmail.com) wrote: > > > > > > Hey there, > > > > > > I have already spent half the day in the irc channel and karmi as well > > > > as clintongormley were really helpful but in the end my example still > > > > does not work as expected. > > > > > > What I want to achieve: > > > > > > I have docs which include an uri field like > > > > > > { uri : "[http://www.foobar.com](http://www.foobar.com)" } > > > > > > And I want to be able to find that doc by searching for "foo" > > > > > > { uri : "[http://www.mylatestwebsite.com](http://www.mylatestwebsite.com)" } > > > > > > And I want to be able to find it by searching for "latest" > > > > > > Now with the standard analyzer / tokenizer it doesn't split the domain > > > > after the dots as there is no whitespace and therefore the dots are > > > > considered as part of the token. Also its quite common for domains to > > > > be not seperated by tokens like the one above. > > > > > > Now collin, karmi and kimchy suggested using ngram for this and collin > > > > even provided an example which unfortunately did not work. So I > > > > produced a minimal example based on collins which I would expect to > > > > work but doesn't and I'd love to hear about any suggestions how to > > > > make this work. > > > > > > My ES Session for this Problem looks like this: > > > > > > [https://gist.github.com/gists/961418](https://gist.github.com/gists/961418) > > > > > > Kind regards, John > > > > > ## -- > > > > > > Paul Loy > > > [p...@keteracel.com](mailto:p...@keteracel.com) > > > [Paul Loy - Amihan Entertainment | LinkedIn](http://uk.linkedin.com/in/paulloy) > > > > ## -- > > > > Paul Loy > > p...@keteracel.comhttp://uk.linkedin.com/in/paulloy ## -- Paul Loy [paul@keteracel.com](mailto:paul@keteracel.com) [http://uk.linkedin.com/in/paulloy](http://uk.linkedin.com/in/paulloy) ---
**Author:** ![hukl](https://avatars.discourse-cdn.com/v4/letter/h/ea5d25/32.png) [@hukl](https://discuss.elastic.co/u/hukl)\ **Post date:** [May 8, 2011, 6:09pm UTC](https://discuss.elastic.co/t/cant-get-ngram-indexing-querying-to-work-as-expected/4363/6 "2011-05-08T18:09:42Z")
Lets add a concrete example: I added two urls: { uri : "[http://www.heise.de](http://www.heise.de)" } { uri : "[http://mylatestwebsite.com](http://mylatestwebsite.com)" } and I'm using the ascii\_ngram analyzer I can search for "hei", "heise", "latest" and always get both results at least they seem to be in proper order. Searching for "latest" returns the mylatestwebsite doc on first position etc. Still I'd expect to get only one result when searching for a substring that is only included in one of the docs. Is ngram even the right thing for that? As you propbably guess - I'm still confused. Kind regards, John On May 8, 7:56 pm, hukl [jpba...@gmail.com](mailto:jpba...@gmail.com) wrote: > Thank you! Indeed this makes sense and returns results but the problem > is that when I enter more urls it will return them all on each query. > > This was also mentioned by collin in this thread:[http://groups.google.com/a/elasticsearch.com/group/users/browse\_threa](http://groups.google.com/a/elasticsearch.com/group/users/browse_threa)... > > where he suggested to use a different search tokenizer. > > So the question would be which search analyzer / tokenizer would fit? > Or is it a bug? > > Kind regards, John > > On May 8, 7:22 pm, Paul Loy [ketera...@gmail.com](mailto:ketera...@gmail.com) wrote: > > > Actually scratch that. What worked was changing the search analyser to > > ascii\_ngram. I think the default n for ngram is min 2, max 3. So since heis > > is 4 characters it will not match any of the tokens in the index. > > > On Sun, May 8, 2011 at 6:15 PM, Paul Loy [ketera...@gmail.com](mailto:ketera...@gmail.com) wrote: > > > > > After a bit of playing I did this: > > > > $ curl -XPUT '[http://127.0.0.1:9200/test/website4/\_mapping?pretty=1'-d'](http://127.0.0.1:9200/test/website4/_mapping?pretty=1'-d') > > > { > > > "website" : { > > > "properties" : { > > > "uri" : { > > > "type" : "string", > > > "include\_in\_all" : 0, > > > "index\_analyzer" : "ascii\_ngram", > > > "search\_analyzer" : "ascii\_std" > > > } > > > } > > > } > > > } > > > ' > > > { > > > "ok" : true, > > > "acknowledged" : true > > > } > > > > $ curl -XPOST '[http://127.0.0.1:9200/test/website4?pretty=1'-d'](http://127.0.0.1:9200/test/website4?pretty=1'-d') > > > { > > > "uri" : "[http://www.heise.de](http://www.heise.de)" > > > } > > > ' > > > { > > > "ok" : true, > > > "\_index" : "test", > > > "\_type" : "website4", > > > "\_id" : "j3KS0Py1TWC7JjnOWeeCIg", > > > "\_version" : 1 > > > } > > > > $ _curl -XPOST '[http://127.0.0.1:9200/test/\_refresh](http://127.0.0.1:9200/test/_refresh)'_ > > > {"ok":true,"\_shards":{"total":10,"successful":5,"failed":0}} > > > > $ curl -XGET '[http://127.0.0.1:9200/test/website4/\_search?pretty=1'-d'](http://127.0.0.1:9200/test/website4/_search?pretty=1'-d') > > > { > > > "query" : { > > > "field" : { > > > "uri" : "heis" > > > } > > > } > > > } > > > ' > > > { > > > "took" : 4, > > > "timed\_out" : false, > > > "\_shards" : { > > > "total" : 5, > > > "successful" : 5, > > > "failed" : 0 > > > }, > > > \*"hits" : { > > > "total" : 1, > > > "max\_score" : 0.28602687, > > > "hits" : [ { > > > "\_index" : "test", > > > "\_type" : "website4", > > > "\_id" : "j3KS0Py1TWC7JjnOWeeCIg", > > > "\_score" : 0.28602687, "\_source" : > > > { > > > "uri" : "[http://www.heise.de](http://www.heise.de)" > > > } > > > > ``` > > > } ] > > > > > > ``` > > > > > > }\* > > > } > > > > Putting a refresh in there works. > > > > On Sun, May 8, 2011 at 4:17 PM, hukl [jpba...@gmail.com](mailto:jpba...@gmail.com) wrote: > > > > > Hey there, > > > > > I have already spent half the day in the irc channel and karmi as well > > > > as clintongormley were really helpful but in the end my example still > > > > does not work as expected. > > > > > What I want to achieve: > > > > > I have docs which include an uri field like > > > > > { uri : "[http://www.foobar.com](http://www.foobar.com)" } > > > > > And I want to be able to find that doc by searching for "foo" > > > > > { uri : "[http://www.mylatestwebsite.com](http://www.mylatestwebsite.com)" } > > > > > And I want to be able to find it by searching for "latest" > > > > > Now with the standard analyzer / tokenizer it doesn't split the domain > > > > after the dots as there is no whitespace and therefore the dots are > > > > considered as part of the token. Also its quite common for domains to > > > > be not seperated by tokens like the one above. > > > > > Now collin, karmi and kimchy suggested using ngram for this and collin > > > > even provided an example which unfortunately did not work. So I > > > > produced a minimal example based on collins which I would expect to > > > > work but doesn't and I'd love to hear about any suggestions how to > > > > make this work. > > > > > My ES Session for this Problem looks like this: > > > > > [https://gist.github.com/gists/961418](https://gist.github.com/gists/961418) > > > > > Kind regards, John > > > > ## -- > > > > > > Paul Loy > > > [p...@keteracel.com](mailto:p...@keteracel.com) > > > [http://uk.linkedin.com/in/paulloy](http://uk.linkedin.com/in/paulloy) > > > ## -- > > > > Paul Loy > > p...@keteracel.comhttp://uk.linkedin.com/in/paulloy ---
**Author:** ![Paul\_Loy](https://avatars.discourse-cdn.com/v4/letter/p/ad7895/32.png) [@Paul\_Loy](https://discuss.elastic.co/u/Paul_Loy)\ **Post date:** [May 8, 2011, 6:10pm UTC](https://discuss.elastic.co/t/cant-get-ngram-indexing-querying-to-work-as-expected/4363/7 "2011-05-08T18:10:14Z")
Hang on, looking at > **[Elastic — The Search AI Company](https://www.elastic.co)** > > Power insights and outcomes with The Elastic Search AI Platform. See into your data and find answers that matter with enterprise solutions designed to help you accelerate time to insight. Try Elastic ... min\_ngram is 1! On Sun, May 8, 2011 at 7:07 PM, Paul Loy [keteracel@gmail.com](mailto:keteracel@gmail.com) wrote: > Yup, you are correct. That's a shame as I was going to use ngram for > username search ☹ > > On Sun, May 8, 2011 at 6:56 PM, hukl [jpbader@gmail.com](mailto:jpbader@gmail.com) wrote: > > > Thank you! Indeed this makes sense and returns results but the problem > > is that when I enter more urls it will return them all on each query. > > > > This was also mentioned by collin in this thread: > > > > [http://groups.google.com/a/elasticsearch.com/group/users/browse\_thread/thread/cd08abb8ba01b0d4/38c0ed74e263252a?lnk=gst&q=ngram#38c0ed74e263252a](http://groups.google.com/a/elasticsearch.com/group/users/browse_thread/thread/cd08abb8ba01b0d4/38c0ed74e263252a?lnk=gst&q=ngram#38c0ed74e263252a) > > > > where he suggested to use a different search tokenizer. > > > > So the question would be which search analyzer / tokenizer would fit? > > Or is it a bug? > > > > Kind regards, John > > > > On May 8, 7:22 pm, Paul Loy [ketera...@gmail.com](mailto:ketera...@gmail.com) wrote: > > > > > Actually scratch that. What worked was changing the search analyser to > > > ascii\_ngram. I think the default n for ngram is min 2, max 3. So since > > > heis > > > is 4 characters it will not match any of the tokens in the index. > > > > > > On Sun, May 8, 2011 at 6:15 PM, Paul Loy [ketera...@gmail.com](mailto:ketera...@gmail.com) wrote: > > > > > > > After a bit of playing I did this: > > > > > > > $ curl -XPUT ' > > > > [http://127.0.0.1:9200/test/website4/\_mapping?pretty=1'-d](http://127.0.0.1:9200/test/website4/_mapping?pretty=1'-d) ' > > > > { > > > > "website" : { > > > > "properties" : { > > > > "uri" : { > > > > "type" : "string", > > > > "include\_in\_all" : 0, > > > > "index\_analyzer" : "ascii\_ngram", > > > > "search\_analyzer" : "ascii\_std" > > > > } > > > > } > > > > } > > > > } > > > > ' > > > > { > > > > "ok" : true, > > > > "acknowledged" : true > > > > } > > > > > > > $ curl -XPOST '[http://127.0.0.1:9200/test/website4?pretty=1'-d](http://127.0.0.1:9200/test/website4?pretty=1'-d) ' > > > > { > > > > "uri" : "[http://www.heise.de](http://www.heise.de)" > > > > } > > > > ' > > > > { > > > > "ok" : true, > > > > "\_index" : "test", > > > > "\_type" : "website4", > > > > "\_id" : "j3KS0Py1TWC7JjnOWeeCIg", > > > > "\_version" : 1 > > > > } > > > > > > > $ _curl -XPOST '[http://127.0.0.1:9200/test/\_refresh](http://127.0.0.1:9200/test/_refresh)'_ > > > > {"ok":true,"\_shards":{"total":10,"successful":5,"failed":0}} > > > > > > > $ curl -XGET '[http://127.0.0.1:9200/test/website4/\_search?pretty=1'-d'](http://127.0.0.1:9200/test/website4/_search?pretty=1'-d') > > > > { > > > > "query" : { > > > > "field" : { > > > > "uri" : "heis" > > > > } > > > > } > > > > } > > > > ' > > > > { > > > > "took" : 4, > > > > "timed\_out" : false, > > > > "\_shards" : { > > > > "total" : 5, > > > > "successful" : 5, > > > > "failed" : 0 > > > > }, > > > > \*"hits" : { > > > > "total" : 1, > > > > "max\_score" : 0.28602687, > > > > "hits" : [ { > > > > "\_index" : "test", > > > > "\_type" : "website4", > > > > "\_id" : "j3KS0Py1TWC7JjnOWeeCIg", > > > > "\_score" : 0.28602687, "\_source" : > > > > { > > > > "uri" : "[http://www.heise.de](http://www.heise.de)" > > > > } > > > > > > > ``` > > > > } ] > > > > > > > > ``` > > > > > > > > }\* > > > > } > > > > > > > Putting a refresh in there works. > > > > > > > On Sun, May 8, 2011 at 4:17 PM, hukl [jpba...@gmail.com](mailto:jpba...@gmail.com) wrote: > > > > > > > > Hey there, > > > > > > > > I have already spent half the day in the irc channel and karmi as > > > > > well > > > > > as clintongormley were really helpful but in the end my example still > > > > > does not work as expected. > > > > > > > > What I want to achieve: > > > > > > > > I have docs which include an uri field like > > > > > > > > { uri : "[http://www.foobar.com](http://www.foobar.com)" } > > > > > > > > And I want to be able to find that doc by searching for "foo" > > > > > > > > { uri : "[http://www.mylatestwebsite.com](http://www.mylatestwebsite.com)" } > > > > > > > > And I want to be able to find it by searching for "latest" > > > > > > > > Now with the standard analyzer / tokenizer it doesn't split the > > > > > domain > > > > > after the dots as there is no whitespace and therefore the dots are > > > > > considered as part of the token. Also its quite common for domains to > > > > > be not seperated by tokens like the one above. > > > > > > > > Now collin, karmi and kimchy suggested using ngram for this and > > > > > collin > > > > > even provided an example which unfortunately did not work. So I > > > > > produced a minimal example based on collins which I would expect to > > > > > work but doesn't and I'd love to hear about any suggestions how to > > > > > make this work. > > > > > > > > My ES Session for this Problem looks like this: > > > > > > > > [https://gist.github.com/gists/961418](https://gist.github.com/gists/961418) > > > > > > > > Kind regards, John > > > > > > > ## -- > > > > > > > > Paul Loy > > > > [p...@keteracel.com](mailto:p...@keteracel.com) > > > > [Paul Loy - Amihan Entertainment | LinkedIn](http://uk.linkedin.com/in/paulloy) > > > > > > ## -- > > > > > > Paul Loy > > > p...@keteracel.comhttp://uk.linkedin.com/in/paulloy > > ## -- > > Paul Loy > [paul@keteracel.com](mailto:paul@keteracel.com) > > [Paul Loy - Amihan Entertainment | LinkedIn](http://uk.linkedin.com/in/paulloy) ## -- Paul Loy [paul@keteracel.com](mailto:paul@keteracel.com) [http://uk.linkedin.com/in/paulloy](http://uk.linkedin.com/in/paulloy) ---
**Author:** ![Paul\_Loy](https://avatars.discourse-cdn.com/v4/letter/p/ad7895/32.png) [@Paul\_Loy](https://discuss.elastic.co/u/Paul_Loy)\ **Post date:** [May 8, 2011, 6:13pm UTC](https://discuss.elastic.co/t/cant-get-ngram-indexing-querying-to-work-as-expected/4363/8 "2011-05-08T18:13:43Z")
So yeah, if you search for zzzz it won't find anything. You need to set min\_gram and max\_gram to different values. On Sun, May 8, 2011 at 7:10 PM, Paul Loy [keteracel@gmail.com](mailto:keteracel@gmail.com) wrote: > Hang on, looking at > [Elastic — The Search AI Company | Elastic](http://www.elasticsearch.org/guide/reference/index-modules/analysis/ngram-tokenizer.htmlthe) min\_ngram is 1! > > On Sun, May 8, 2011 at 7:07 PM, Paul Loy [keteracel@gmail.com](mailto:keteracel@gmail.com) wrote: > > > Yup, you are correct. That's a shame as I was going to use ngram for > > username search ☹ > > > > On Sun, May 8, 2011 at 6:56 PM, hukl [jpbader@gmail.com](mailto:jpbader@gmail.com) wrote: > > > > > Thank you! Indeed this makes sense and returns results but the problem > > > is that when I enter more urls it will return them all on each query. > > > > > > This was also mentioned by collin in this thread: > > > > > > [http://groups.google.com/a/elasticsearch.com/group/users/browse\_thread/thread/cd08abb8ba01b0d4/38c0ed74e263252a?lnk=gst&q=ngram#38c0ed74e263252a](http://groups.google.com/a/elasticsearch.com/group/users/browse_thread/thread/cd08abb8ba01b0d4/38c0ed74e263252a?lnk=gst&q=ngram#38c0ed74e263252a) > > > > > > where he suggested to use a different search tokenizer. > > > > > > So the question would be which search analyzer / tokenizer would fit? > > > Or is it a bug? > > > > > > Kind regards, John > > > > > > On May 8, 7:22 pm, Paul Loy [ketera...@gmail.com](mailto:ketera...@gmail.com) wrote: > > > > > > > Actually scratch that. What worked was changing the search analyser to > > > > ascii\_ngram. I think the default n for ngram is min 2, max 3. So since > > > > heis > > > > is 4 characters it will not match any of the tokens in the index. > > > > > > > > On Sun, May 8, 2011 at 6:15 PM, Paul Loy [ketera...@gmail.com](mailto:ketera...@gmail.com) wrote: > > > > > > > > > After a bit of playing I did this: > > > > > > > > > $ curl -XPUT ' > > > > > [http://127.0.0.1:9200/test/website4/\_mapping?pretty=1'-d](http://127.0.0.1:9200/test/website4/_mapping?pretty=1'-d) ' > > > > > { > > > > > "website" : { > > > > > "properties" : { > > > > > "uri" : { > > > > > "type" : "string", > > > > > "include\_in\_all" : 0, > > > > > "index\_analyzer" : "ascii\_ngram", > > > > > "search\_analyzer" : "ascii\_std" > > > > > } > > > > > } > > > > > } > > > > > } > > > > > ' > > > > > { > > > > > "ok" : true, > > > > > "acknowledged" : true > > > > > } > > > > > > > > > $ curl -XPOST '[http://127.0.0.1:9200/test/website4?pretty=1'-d](http://127.0.0.1:9200/test/website4?pretty=1'-d) ' > > > > > { > > > > > "uri" : "[http://www.heise.de](http://www.heise.de)" > > > > > } > > > > > ' > > > > > { > > > > > "ok" : true, > > > > > "\_index" : "test", > > > > > "\_type" : "website4", > > > > > "\_id" : "j3KS0Py1TWC7JjnOWeeCIg", > > > > > "\_version" : 1 > > > > > } > > > > > > > > > $ _curl -XPOST '[http://127.0.0.1:9200/test/\_refresh](http://127.0.0.1:9200/test/_refresh)'_ > > > > > {"ok":true,"\_shards":{"total":10,"successful":5,"failed":0}} > > > > > > > > > $ curl -XGET ' > > > > > [http://127.0.0.1:9200/test/website4/\_search?pretty=1'-d](http://127.0.0.1:9200/test/website4/_search?pretty=1'-d) ' > > > > > { > > > > > "query" : { > > > > > "field" : { > > > > > "uri" : "heis" > > > > > } > > > > > } > > > > > } > > > > > ' > > > > > { > > > > > "took" : 4, > > > > > "timed\_out" : false, > > > > > "\_shards" : { > > > > > "total" : 5, > > > > > "successful" : 5, > > > > > "failed" : 0 > > > > > }, > > > > > \*"hits" : { > > > > > "total" : 1, > > > > > "max\_score" : 0.28602687, > > > > > "hits" : [ { > > > > > "\_index" : "test", > > > > > "\_type" : "website4", > > > > > "\_id" : "j3KS0Py1TWC7JjnOWeeCIg", > > > > > "\_score" : 0.28602687, "\_source" : > > > > > { > > > > > "uri" : "[http://www.heise.de](http://www.heise.de)" > > > > > } > > > > > > > > > ``` > > > > > } ] > > > > > > > > > > ``` > > > > > > > > > > }\* > > > > > } > > > > > > > > > Putting a refresh in there works. > > > > > > > > > On Sun, May 8, 2011 at 4:17 PM, hukl [jpba...@gmail.com](mailto:jpba...@gmail.com) wrote: > > > > > > > > > > Hey there, > > > > > > > > > > I have already spent half the day in the irc channel and karmi as > > > > > > well > > > > > > as clintongormley were really helpful but in the end my example > > > > > > still > > > > > > does not work as expected. > > > > > > > > > > What I want to achieve: > > > > > > > > > > I have docs which include an uri field like > > > > > > > > > > { uri : "[http://www.foobar.com](http://www.foobar.com)" } > > > > > > > > > > And I want to be able to find that doc by searching for "foo" > > > > > > > > > > { uri : "[http://www.mylatestwebsite.com](http://www.mylatestwebsite.com)" } > > > > > > > > > > And I want to be able to find it by searching for "latest" > > > > > > > > > > Now with the standard analyzer / tokenizer it doesn't split the > > > > > > domain > > > > > > after the dots as there is no whitespace and therefore the dots are > > > > > > considered as part of the token. Also its quite common for domains > > > > > > to > > > > > > be not seperated by tokens like the one above. > > > > > > > > > > Now collin, karmi and kimchy suggested using ngram for this and > > > > > > collin > > > > > > even provided an example which unfortunately did not work. So I > > > > > > produced a minimal example based on collins which I would expect to > > > > > > work but doesn't and I'd love to hear about any suggestions how to > > > > > > make this work. > > > > > > > > > > My ES Session for this Problem looks like this: > > > > > > > > > > [https://gist.github.com/gists/961418](https://gist.github.com/gists/961418) > > > > > > > > > > Kind regards, John > > > > > > > > > ## -- > > > > > > > > > > Paul Loy > > > > > [p...@keteracel.com](mailto:p...@keteracel.com) > > > > > [Paul Loy - Amihan Entertainment | LinkedIn](http://uk.linkedin.com/in/paulloy) > > > > > > > > ## -- > > > > > > > > Paul Loy > > > > p...@keteracel.comhttp://uk.linkedin.com/in/paulloy > > > > ## -- > > > > Paul Loy > > [paul@keteracel.com](mailto:paul@keteracel.com) > > > > [Paul Loy - Amihan Entertainment | LinkedIn](http://uk.linkedin.com/in/paulloy) > > ## -- > > Paul Loy > [paul@keteracel.com](mailto:paul@keteracel.com) > [Paul Loy - Amihan Entertainment | LinkedIn](http://uk.linkedin.com/in/paulloy) ## -- Paul Loy [paul@keteracel.com](mailto:paul@keteracel.com) [http://uk.linkedin.com/in/paulloy](http://uk.linkedin.com/in/paulloy) ---
**Author:** ![hukl](https://avatars.discourse-cdn.com/v4/letter/h/ea5d25/32.png) [@hukl](https://discuss.elastic.co/u/hukl)\ **Post date:** [May 8, 2011, 6:15pm UTC](https://discuss.elastic.co/t/cant-get-ngram-indexing-querying-to-work-as-expected/4363/9 "2011-05-08T18:15:28Z")
Any suggestions for proper values ? 😉 On May 8, 8:13 pm, Paul Loy [ketera...@gmail.com](mailto:ketera...@gmail.com) wrote: > So yeah, if you search for zzzz it won't find anything. You need to set > min\_gram and max\_gram to different values. > > On Sun, May 8, 2011 at 7:10 PM, Paul Loy [ketera...@gmail.com](mailto:ketera...@gmail.com) wrote: > > > Hang on, looking at > > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/index-modules/analysis/n...min_ngram) is 1! > > > On Sun, May 8, 2011 at 7:07 PM, Paul Loy [ketera...@gmail.com](mailto:ketera...@gmail.com) wrote: > > > > Yup, you are correct. That's a shame as I was going to use ngram for > > > username search ☹ > > > > On Sun, May 8, 2011 at 6:56 PM, hukl [jpba...@gmail.com](mailto:jpba...@gmail.com) wrote: > > > > > Thank you! Indeed this makes sense and returns results but the problem > > > > is that when I enter more urls it will return them all on each query. > > > > > This was also mentioned by collin in this thread: > > > > > [http://groups.google.com/a/elasticsearch.com/group/users/browse\_threa](http://groups.google.com/a/elasticsearch.com/group/users/browse_threa)... > > > > > where he suggested to use a different search tokenizer. > > > > > So the question would be which search analyzer / tokenizer would fit? > > > > Or is it a bug? > > > > > Kind regards, John > > > > > On May 8, 7:22 pm, Paul Loy [ketera...@gmail.com](mailto:ketera...@gmail.com) wrote: > > > > > > > > > Actually scratch that. What worked was changing the search analyser to > > > > > ascii\_ngram. I think the default n for ngram is min 2, max 3. So since > > > > > heis > > > > > is 4 characters it will not match any of the tokens in the index. > > > > > > On Sun, May 8, 2011 at 6:15 PM, Paul Loy [ketera...@gmail.com](mailto:ketera...@gmail.com) wrote: > > > > > > > > > > > After a bit of playing I did this: > > > > > > > $ curl -XPUT ' > > > > > > [http://127.0.0.1:9200/test/website4/\_mapping?pretty=1'-d'](http://127.0.0.1:9200/test/website4/_mapping?pretty=1'-d') > > > > > > { > > > > > > "website" : { > > > > > > "properties" : { > > > > > > "uri" : { > > > > > > "type" : "string", > > > > > > "include\_in\_all" : 0, > > > > > > "index\_analyzer" : "ascii\_ngram", > > > > > > "search\_analyzer" : "ascii\_std" > > > > > > } > > > > > > } > > > > > > } > > > > > > } > > > > > > ' > > > > > > { > > > > > > "ok" : true, > > > > > > "acknowledged" : true > > > > > > } > > > > > > > $ curl -XPOST '[http://127.0.0.1:9200/test/website4?pretty=1'-d'](http://127.0.0.1:9200/test/website4?pretty=1'-d') > > > > > > { > > > > > > "uri" : "[http://www.heise.de](http://www.heise.de)" > > > > > > } > > > > > > ' > > > > > > { > > > > > > "ok" : true, > > > > > > "\_index" : "test", > > > > > > "\_type" : "website4", > > > > > > "\_id" : "j3KS0Py1TWC7JjnOWeeCIg", > > > > > > "\_version" : 1 > > > > > > } > > > > > > > $ _curl -XPOST '[http://127.0.0.1:9200/test/\_refresh](http://127.0.0.1:9200/test/_refresh)'_ > > > > > > {"ok":true,"\_shards":{"total":10,"successful":5,"failed":0}} > > > > > > > $ curl -XGET ' > > > > > > [http://127.0.0.1:9200/test/website4/\_search?pretty=1'-d'](http://127.0.0.1:9200/test/website4/_search?pretty=1'-d') > > > > > > { > > > > > > "query" : { > > > > > > "field" : { > > > > > > "uri" : "heis" > > > > > > } > > > > > > } > > > > > > } > > > > > > ' > > > > > > { > > > > > > "took" : 4, > > > > > > "timed\_out" : false, > > > > > > "\_shards" : { > > > > > > "total" : 5, > > > > > > "successful" : 5, > > > > > > "failed" : 0 > > > > > > }, > > > > > > \*"hits" : { > > > > > > "total" : 1, > > > > > > "max\_score" : 0.28602687, > > > > > > "hits" : [ { > > > > > > "\_index" : "test", > > > > > > "\_type" : "website4", > > > > > > "\_id" : "j3KS0Py1TWC7JjnOWeeCIg", > > > > > > "\_score" : 0.28602687, "\_source" : > > > > > > { > > > > > > "uri" : "[http://www.heise.de](http://www.heise.de)" > > > > > > } > > > > > > > ``` > > > > > > } ] > > > > > > > > > > > > ``` > > > > > > > > > > > > }\* > > > > > > } > > > > > > > Putting a refresh in there works. > > > > > > > On Sun, May 8, 2011 at 4:17 PM, hukl [jpba...@gmail.com](mailto:jpba...@gmail.com) wrote: > > > > > > > > Hey there, > > > > > > > > I have already spent half the day in the irc channel and karmi as > > > > > > > well > > > > > > > as clintongormley were really helpful but in the end my example > > > > > > > still > > > > > > > does not work as expected. > > > > > > > > What I want to achieve: > > > > > > > > I have docs which include an uri field like > > > > > > > > { uri : "[http://www.foobar.com](http://www.foobar.com)" } > > > > > > > > And I want to be able to find that doc by searching for "foo" > > > > > > > > { uri : "[http://www.mylatestwebsite.com](http://www.mylatestwebsite.com)" } > > > > > > > > And I want to be able to find it by searching for "latest" > > > > > > > > Now with the standard analyzer / tokenizer it doesn't split the > > > > > > > domain > > > > > > > after the dots as there is no whitespace and therefore the dots are > > > > > > > considered as part of the token. Also its quite common for domains > > > > > > > to > > > > > > > be not seperated by tokens like the one above. > > > > > > > > Now collin, karmi and kimchy suggested using ngram for this and > > > > > > > collin > > > > > > > even provided an example which unfortunately did not work. So I > > > > > > > produced a minimal example based on collins which I would expect to > > > > > > > work but doesn't and I'd love to hear about any suggestions how to > > > > > > > make this work. > > > > > > > > My ES Session for this Problem looks like this: > > > > > > > > [https://gist.github.com/gists/961418](https://gist.github.com/gists/961418) > > > > > > > > Kind regards, John > > > > > > > ## -- > > > > > > > > > > > > Paul Loy > > > > > > [p...@keteracel.com](mailto:p...@keteracel.com) > > > > > > [http://uk.linkedin.com/in/paulloy](http://uk.linkedin.com/in/paulloy) > > > > > > ## -- > > > > > > > > > > Paul Loy > > > > > p...@keteracel.comhttp://uk.linkedin.com/in/paulloy > > > > ## -- > > > > > > Paul Loy > > > [p...@keteracel.com](mailto:p...@keteracel.com) > > > > [http://uk.linkedin.com/in/paulloy](http://uk.linkedin.com/in/paulloy) > > > ## -- > > > > Paul Loy > > [p...@keteracel.com](mailto:p...@keteracel.com) > > [http://uk.linkedin.com/in/paulloy](http://uk.linkedin.com/in/paulloy) > > ## -- > > Paul Loy > p...@keteracel.comhttp://uk.linkedin.com/in/paulloy ---
**Author:** ![Paul\_Loy](https://avatars.discourse-cdn.com/v4/letter/p/ad7895/32.png) [@Paul\_Loy](https://discuss.elastic.co/u/Paul_Loy)\ **Post date:** [May 8, 2011, 6:19pm UTC](https://discuss.elastic.co/t/cant-get-ngram-indexing-querying-to-work-as-expected/4363/10 "2011-05-08T18:19:08Z")
I was going to go for 3,5 myself. On Sun, May 8, 2011 at 7:15 PM, hukl [jpbader@gmail.com](mailto:jpbader@gmail.com) wrote: > Any suggestions for proper values ? 😉 > > On May 8, 8:13 pm, Paul Loy [ketera...@gmail.com](mailto:ketera...@gmail.com) wrote: > > > So yeah, if you search for zzzz it won't find anything. You need to set > > min\_gram and max\_gram to different values. > > > > On Sun, May 8, 2011 at 7:10 PM, Paul Loy [ketera...@gmail.com](mailto:ketera...@gmail.com) wrote: > > > > > Hang on, looking at > > [Elastic — The Search AI Company | Elastic](http://www.elasticsearch.org/guide/reference/index-modules/analysis/n...min_ngramis) 1! > > > > On Sun, May 8, 2011 at 7:07 PM, Paul Loy [ketera...@gmail.com](mailto:ketera...@gmail.com) wrote: > > > > > > Yup, you are correct. That's a shame as I was going to use ngram for > > > > username search ☹ > > > > > > On Sun, May 8, 2011 at 6:56 PM, hukl [jpba...@gmail.com](mailto:jpba...@gmail.com) wrote: > > > > > > > Thank you! Indeed this makes sense and returns results but the > > > > > problem > > > > > is that when I enter more urls it will return them all on each query. > > > > > > > This was also mentioned by collin in this thread: > > > > > > > > > [http://groups.google.com/a/elasticsearch.com/group/users/browse\_threa](http://groups.google.com/a/elasticsearch.com/group/users/browse_threa)... > > > > > > where he suggested to use a different search tokenizer. > > > > > > > So the question would be which search analyzer / tokenizer would fit? > > > > > Or is it a bug? > > > > > > > Kind regards, John > > > > > > > On May 8, 7:22 pm, Paul Loy [ketera...@gmail.com](mailto:ketera...@gmail.com) wrote: > > > > > > > > > > > Actually scratch that. What worked was changing the search analyser > > > > > > to > > > > > > ascii\_ngram. I think the default n for ngram is min 2, max 3. So > > > > > > since > > > > > > heis > > > > > > is 4 characters it will not match any of the tokens in the index. > > > > > > > > On Sun, May 8, 2011 at 6:15 PM, Paul Loy [ketera...@gmail.com](mailto:ketera...@gmail.com) > > > > > > wrote: > > > > > > > > > > > > > After a bit of playing I did this: > > > > > > > > > $ curl -XPUT ' > > > > > > > [http://127.0.0.1:9200/test/website4/\_mapping?pretty=1'-d'](http://127.0.0.1:9200/test/website4/_mapping?pretty=1'-d') > > > > > > > { > > > > > > > "website" : { > > > > > > > "properties" : { > > > > > > > "uri" : { > > > > > > > "type" : "string", > > > > > > > "include\_in\_all" : 0, > > > > > > > "index\_analyzer" : "ascii\_ngram", > > > > > > > "search\_analyzer" : "ascii\_std" > > > > > > > } > > > > > > > } > > > > > > > } > > > > > > > } > > > > > > > ' > > > > > > > { > > > > > > > "ok" : true, > > > > > > > "acknowledged" : true > > > > > > > } > > > > > > > > > $ curl -XPOST '[http://127.0.0.1:9200/test/website4?pretty=1'-d'](http://127.0.0.1:9200/test/website4?pretty=1'-d') > > > > > > > { > > > > > > > "uri" : "[http://www.heise.de](http://www.heise.de)" > > > > > > > } > > > > > > > ' > > > > > > > { > > > > > > > "ok" : true, > > > > > > > "\_index" : "test", > > > > > > > "\_type" : "website4", > > > > > > > "\_id" : "j3KS0Py1TWC7JjnOWeeCIg", > > > > > > > "\_version" : 1 > > > > > > > } > > > > > > > > > $ _curl -XPOST '[http://127.0.0.1:9200/test/\_refresh](http://127.0.0.1:9200/test/_refresh)'_ > > > > > > > {"ok":true,"\_shards":{"total":10,"successful":5,"failed":0}} > > > > > > > > > $ curl -XGET ' > > > > > > > [http://127.0.0.1:9200/test/website4/\_search?pretty=1'-d'](http://127.0.0.1:9200/test/website4/_search?pretty=1'-d') > > > > > > > { > > > > > > > "query" : { > > > > > > > "field" : { > > > > > > > "uri" : "heis" > > > > > > > } > > > > > > > } > > > > > > > } > > > > > > > ' > > > > > > > { > > > > > > > "took" : 4, > > > > > > > "timed\_out" : false, > > > > > > > "\_shards" : { > > > > > > > "total" : 5, > > > > > > > "successful" : 5, > > > > > > > "failed" : 0 > > > > > > > }, > > > > > > > \*"hits" : { > > > > > > > "total" : 1, > > > > > > > "max\_score" : 0.28602687, > > > > > > > "hits" : [ { > > > > > > > "\_index" : "test", > > > > > > > "\_type" : "website4", > > > > > > > "\_id" : "j3KS0Py1TWC7JjnOWeeCIg", > > > > > > > "\_score" : 0.28602687, "\_source" : > > > > > > > { > > > > > > > "uri" : "[http://www.heise.de](http://www.heise.de)" > > > > > > > } > > > > > > > > > ``` > > > > > > > } ] > > > > > > > > > > > > > > ``` > > > > > > > > > > > > > > }\* > > > > > > > } > > > > > > > > > Putting a refresh in there works. > > > > > > > > > On Sun, May 8, 2011 at 4:17 PM, hukl [jpba...@gmail.com](mailto:jpba...@gmail.com) wrote: > > > > > > > > > > Hey there, > > > > > > > > > > I have already spent half the day in the irc channel and karmi > > > > > > > > as > > > > > > > > well > > > > > > > > as clintongormley were really helpful but in the end my example > > > > > > > > still > > > > > > > > does not work as expected. > > > > > > > > > > What I want to achieve: > > > > > > > > > > I have docs which include an uri field like > > > > > > > > > > { uri : "[http://www.foobar.com](http://www.foobar.com)" } > > > > > > > > > > And I want to be able to find that doc by searching for "foo" > > > > > > > > > > { uri : "[http://www.mylatestwebsite.com](http://www.mylatestwebsite.com)" } > > > > > > > > > > And I want to be able to find it by searching for "latest" > > > > > > > > > > Now with the standard analyzer / tokenizer it doesn't split the > > > > > > > > domain > > > > > > > > after the dots as there is no whitespace and therefore the dots > > > > > > > > are > > > > > > > > considered as part of the token. Also its quite common for > > > > > > > > domains > > > > > > > > to > > > > > > > > be not seperated by tokens like the one above. > > > > > > > > > > Now collin, karmi and kimchy suggested using ngram for this and > > > > > > > > collin > > > > > > > > even provided an example which unfortunately did not work. So I > > > > > > > > produced a minimal example based on collins which I would expect > > > > > > > > to > > > > > > > > work but doesn't and I'd love to hear about any suggestions how > > > > > > > > to > > > > > > > > make this work. > > > > > > > > > > My ES Session for this Problem looks like this: > > > > > > > > > > [https://gist.github.com/gists/961418](https://gist.github.com/gists/961418) > > > > > > > > > > Kind regards, John > > > > > > > > > ## -- > > > > > > > > > > > > > > Paul Loy > > > > > > > [p...@keteracel.com](mailto:p...@keteracel.com) > > > > > > > [Paul Loy - Amihan Entertainment | LinkedIn](http://uk.linkedin.com/in/paulloy) > > > > > > > > ## -- > > > > > > > > > > > > Paul Loy > > > > > > p...@keteracel.comhttp://uk.linkedin.com/in/paulloy > > > > > > ## -- > > > > > > > > Paul Loy > > > > [p...@keteracel.com](mailto:p...@keteracel.com) > > > > > > [Paul Loy - Amihan Entertainment | LinkedIn](http://uk.linkedin.com/in/paulloy) > > > > > ## -- > > > > > > Paul Loy > > > [p...@keteracel.com](mailto:p...@keteracel.com) > > > [Paul Loy - Amihan Entertainment | LinkedIn](http://uk.linkedin.com/in/paulloy) > > > > ## -- > > > > Paul Loy > > p...@keteracel.comhttp://uk.linkedin.com/in/paulloy ## -- Paul Loy [paul@keteracel.com](mailto:paul@keteracel.com) [http://uk.linkedin.com/in/paulloy](http://uk.linkedin.com/in/paulloy) ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 4:06am UTC](https://discuss.elastic.co/t/cant-get-ngram-indexing-querying-to-work-as-expected/4363/11 "2017-07-06T04:06:42Z")