Confused about query_string and the use of wildcards

Clinton_Gormley · March 16, 2011, 3:29pm

Hi Enrique

All my confusion is about the expected behaviour of the wildcard
queries. So let's say, if my user wants to search for "iPhone" and I
run a query with wildcards "iPhone" then:

The wildcards should work as you expect them to work - from your
example, you expect them to find any word containing "iPhone", and that
is correct.

As it is working now, it will not analyze the search term, but just
use iPhone as the token itself, therefore not finding "iPhone" which
has a token of "iphon".

If it weren't analyzed, then the term resulting from "iPhone" would be
"iPhone" not "iphone" so even case would make a difference.

As I expected it to be, i.e. the "iPhone" is analyzed into
"iphon" and then executed the search, and "iPhone" results are
returned.

This is the correct behaviour.

Try it out without the snowball stemmer:

gist.github.com

https://gist.github.com/clintongormley/fa32a01ff47b2c2377d9

gistfile1.txt

# [Wed Mar 16 16:26:13 2011] Protocol: http, Server: 192.168.5.103:9200
curl -XPUT 'http://127.0.0.1:9200/test/?pretty=1' 

# [Wed Mar 16 16:26:13 2011] Response:
# {
#    "ok" : true,
#    "acknowledged" : true
# }

# [Wed Mar 16 16:26:16 2011] Protocol: http, Server: 192.168.5.103:9200

This file has been truncated. show original

You'll see that it works as you expect.

clint

Enrique_Medina_Monte · March 16, 2011, 3:35pm

Try:

{
"query" : {
"field" : {
"_all" : "iphone"
}
}
}

and you'll see that it doesn't work either...

On Wed, Mar 16, 2011 at 4:29 PM, Clinton Gormley clinton@iannounce.co.ukwrote:

Hi Enrique

All my confusion is about the expected behaviour of the wildcard
queries. So let's say, if my user wants to search for "iPhone" and I
run a query with wildcards "iPhone" then:

The wildcards should work as you expect them to work - from your
example, you expect them to find any word containing "iPhone", and that
is correct.

As it is working now, it will not analyze the search term, but just
use iPhone as the token itself, therefore not finding "iPhone" which
has a token of "iphon".

If it weren't analyzed, then the term resulting from "iPhone" would be
"iPhone" not "iphone" so even case would make a difference.

As I expected it to be, i.e. the "iPhone" is analyzed into
"iphon" and then executed the search, and "iPhone" results are
returned.

This is the correct behaviour.

Try it out without the snowball stemmer:

gist:fa32a01ff47b2c2377d9 · GitHub

You'll see that it works as you expect.

clint

Clinton_Gormley · March 16, 2011, 3:37pm

{
"query" : {
"field" : {
"_all" : "iphone"
}
}
}

and you'll see that it doesn't work either...

It works for me:

[Wed Mar 16 16:36:17 2011] Protocol: http, Server: 192.168.5.103:9200

curl -XGET 'http://127.0.0.1:9200/test/_search?pretty=1' -d '
{
"query" : {
"field" : {
"_all" : "iphone"
}
}
}
'

[Wed Mar 16 16:36:17 2011] Response:

{

"hits" : {

"hits" : [

{

"_source" : {

"text" : "I have in iPhone"

},

"_score" : 1,

"_index" : "test",

"_id" : "O9tKwG64Td2_lRhfBm_jag",

"_type" : "doc"

}

],

"max_score" : 1,

"total" : 1

},

"timed_out" : false,

"_shards" : {

"failed" : 0,

"successful" : 5,

"total" : 5

},

"took" : 2

}

What version of ES are you using? What other settings do you have?

clint

Enrique_Medina_Monte · March 16, 2011, 3:46pm

This is ES 0.16.0.SNAPSHOT, and IMHO your example works because you didn't
use a Spanish analyzer, but the standard one, which basically generates same
tokens as words (it doesn't stem as Spanish does).

On Wed, Mar 16, 2011 at 4:37 PM, Clinton Gormley clinton@iannounce.co.ukwrote:

{
"query" : {
"field" : {
"_all" : "iphone"
}
}
}

and you'll see that it doesn't work either...

It works for me:

[Wed Mar 16 16:36:17 2011] Protocol: http, Server: 192.168.5.103:9200

curl -XGET 'http://127.0.0.1:9200/test/_search?pretty=1' -d '
{
"query" : {
"field" : {
"_all" : "iphone"
}
}
}
'

[Wed Mar 16 16:36:17 2011] Response:

{

"hits" : {

"hits" : [

{

"_source" : {

"text" : "I have in iPhone"

},

"_score" : 1,

"_index" : "test",

"_id" : "O9tKwG64Td2_lRhfBm_jag",

"_type" : "doc"

}

],

"max_score" : 1,

"total" : 1

},

"timed_out" : false,

"_shards" : {

"failed" : 0,

"successful" : 5,

"total" : 5

},

"took" : 2

}

What version of ES are you using? What other settings do you have?

clint

Enrique_Medina_Monte · March 16, 2011, 3:55pm

The more I think about this the more I get convinced that the issue is with
analyzers using stemmers, and the fact that the search ignores the stemmer
when parsing the query terms...

On Wed, Mar 16, 2011 at 4:46 PM, Enrique Medina Montenegro <
e.medina.m@gmail.com> wrote:

This is ES 0.16.0.SNAPSHOT, and IMHO your example works because you didn't
use a Spanish analyzer, but the standard one, which basically generates same
tokens as words (it doesn't stem as Spanish does).

On Wed, Mar 16, 2011 at 4:37 PM, Clinton Gormley clinton@iannounce.co.ukwrote:

{
"query" : {
"field" : {
"_all" : "iphone"
}
}
}

and you'll see that it doesn't work either...

It works for me:

[Wed Mar 16 16:36:17 2011] Protocol: http, Server: 192.168.5.103:9200

curl -XGET 'http://127.0.0.1:9200/test/_search?pretty=1' -d '
{
"query" : {
"field" : {
"_all" : "iphone"
}
}
}
'

[Wed Mar 16 16:36:17 2011] Response:

{

"hits" : {

"hits" : [

{

"_source" : {

"text" : "I have in iPhone"

},

"_score" : 1,

"_index" : "test",

"_id" : "O9tKwG64Td2_lRhfBm_jag",

"_type" : "doc"

}

],

"max_score" : 1,

"total" : 1

},

"timed_out" : false,

"_shards" : {

"failed" : 0,

"successful" : 5,

"total" : 5

},

"took" : 2

}

What version of ES are you using? What other settings do you have?

clint

Clinton_Gormley · March 16, 2011, 3:56pm

On Wed, 2011-03-16 at 16:55 +0100, Enrique Medina Montenegro wrote:

The more I think about this the more I get convinced that the issue is
with analyzers using stemmers, and the fact that the search ignores
the stemmer when parsing the query terms...

Exactly. That's why I said there is a bug.

clint

Enrique_Medina_Monte · March 16, 2011, 3:59pm

And just to make it more clear, the bug would be ONLY when using
wildcards...

But I'm afraid this is a Lucene bug, but Shay can comment on it in the
issue...

In the meantime, a workaround is to analyze the search words manually before
composing the query, which is easy to implement but adds some hassle to the
process, obviously.

On Wed, Mar 16, 2011 at 4:56 PM, Clinton Gormley clinton@iannounce.co.ukwrote:

On Wed, 2011-03-16 at 16:55 +0100, Enrique Medina Montenegro wrote:

The more I think about this the more I get convinced that the issue is
with analyzers using stemmers, and the fact that the search ignores
the stemmer when parsing the query terms...

Exactly. That's why I said there is a bug.

clint

kimchy · March 16, 2011, 5:40pm

Hi,

Yea, wildcard text does not get analyzed when used in a query string. This is for the simple reason that analysis might produce more than one token and then it can be confusing as to where to apply the wildcard element.

Having said that, I think I saw somewhere a query parser that did analysis on wildcard. I need to find it. Maybe it can be added as an option.

-shay.banon
On Wednesday, March 16, 2011 at 5:59 PM, Enrique Medina Montenegro wrote:

And just to make it more clear, the bug would be ONLY when using wildcards...

But I'm afraid this is a Lucene bug, but Shay can comment on it in the issue...

In the meantime, a workaround is to analyze the search words manually before composing the query, which is easy to implement but adds some hassle to the process, obviously.

On Wed, Mar 16, 2011 at 4:56 PM, Clinton Gormley clinton@iannounce.co.uk wrote:

On Wed, 2011-03-16 at 16:55 +0100, Enrique Medina Montenegro wrote:

The more I think about this the more I get convinced that the issue is
with analyzers using stemmers, and the fact that the search ignores
the stemmer when parsing the query terms...

Exactly. That's why I said there is a bug.

clint

Clinton_Gormley · March 16, 2011, 5:44pm

Yea, wildcard text does not get analyzed when used in a query
string. This is for the simple reason that analysis might produce more
than one token and then it can be confusing as to where to apply the
wildcard element.

Gaah - so after all that, I was wrong!

Apologies Enrique

clint

Enrique_Medina_Monte · March 16, 2011, 5:50pm

No apologies, that's what users' lists are made for... discussion, right?

In any case, I think that the workaround explained below in this thread
could work fine if in need of stemmizing a wildcard search

On Wed, Mar 16, 2011 at 6:44 PM, Clinton Gormley clinton@iannounce.co.ukwrote:

Yea, wildcard text does not get analyzed when used in a query
string. This is for the simple reason that analysis might produce more
than one token and then it can be confusing as to where to apply the
wildcard element.

Gaah - so after all that, I was wrong!

Apologies Enrique

clint

Enrique_Medina_Monte · March 16, 2011, 5:51pm

Shay,

What would be an example of a word producing more than one token?

Thanks.

On Wed, Mar 16, 2011 at 6:40 PM, Shay Banon shay.banon@elasticsearch.comwrote:

Hi,

Yea, wildcard text does not get analyzed when used in a query string.
This is for the simple reason that analysis might produce more than one
token and then it can be confusing as to where to apply the wildcard
element.

Having said that, I think I saw somewhere a query parser that did
analysis on wildcard. I need to find it. Maybe it can be added as an option.

-shay.banon

On Wednesday, March 16, 2011 at 5:59 PM, Enrique Medina Montenegro wrote:

And just to make it more clear, the bug would be ONLY when using
wildcards...

But I'm afraid this is a Lucene bug, but Shay can comment on it in the
issue...

In the meantime, a workaround is to analyze the search words manually
before composing the query, which is easy to implement but adds some hassle
to the process, obviously.

On Wed, Mar 16, 2011 at 4:56 PM, Clinton Gormley clinton@iannounce.co.ukwrote:

On Wed, 2011-03-16 at 16:55 +0100, Enrique Medina Montenegro wrote:

The more I think about this the more I get convinced that the issue is
with analyzers using stemmers, and the fact that the search ignores
the stemmer when parsing the query terms...

Exactly. That's why I said there is a bug.

clint

kimchy · March 16, 2011, 7:46pm

It really depends on the analyzer you use, and because its unknown on the query parser level, I guess it is not supported. There are analyzers that can result in multiple tokens, like ngram ones.
On Wednesday, March 16, 2011 at 7:51 PM, Enrique Medina Montenegro wrote:

Shay,

What would be an example of a word producing more than one token?

Thanks.

On Wed, Mar 16, 2011 at 6:40 PM, Shay Banon shay.banon@elasticsearch.com wrote:

Hi,

Yea, wildcard text does not get analyzed when used in a query string. This is for the simple reason that analysis might produce more than one token and then it can be confusing as to where to apply the wildcard element.

Having said that, I think I saw somewhere a query parser that did analysis on wildcard. I need to find it. Maybe it can be added as an option.

-shay.banon
On Wednesday, March 16, 2011 at 5:59 PM, Enrique Medina Montenegro wrote:

And just to make it more clear, the bug would be ONLY when using wildcards...

But I'm afraid this is a Lucene bug, but Shay can comment on it in the issue...

In the meantime, a workaround is to analyze the search words manually before composing the query, which is easy to implement but adds some hassle to the process, obviously.

On Wed, Mar 16, 2011 at 4:56 PM, Clinton Gormley clinton@iannounce.co.uk wrote:

On Wed, 2011-03-16 at 16:55 +0100, Enrique Medina Montenegro wrote:

The more I think about this the more I get convinced that the issue is
with analyzers using stemmers, and the fact that the search ignores
the stemmer when parsing the query terms...

Exactly. That's why I said there is a bug.

clint

Enrique_Medina_Monte · March 17, 2011, 9:06am

As commented below in this thread, I eventually went for the workaround of
analyzing the search terms before enclosing them between wildcards and
executing the 'query_string'. As my analyzer is a simple spanish snowball, I
guess I shouldn't get more than 1 token per word, but in any case, my code
would only pick up the first one.

So my question is: would this be a good feature to include in ES? Basically
there could be a property/parameter in the 'query_string' (and even in the
'wildcard' query) that would tell ES to analyze the search terms before
querying when they are enclosed between wildcards (or just have left or
right wildcard). By default it would be set to false, so regular behaviour
applies.

On Wed, Mar 16, 2011 at 8:46 PM, Shay Banon shay.banon@elasticsearch.comwrote:

It really depends on the analyzer you use, and because its unknown on the
query parser level, I guess it is not supported. There are analyzers that
can result in multiple tokens, like ngram ones.

On Wednesday, March 16, 2011 at 7:51 PM, Enrique Medina Montenegro wrote:

Shay,

What would be an example of a word producing more than one token?

Thanks.

On Wed, Mar 16, 2011 at 6:40 PM, Shay Banon shay.banon@elasticsearch.comwrote:

Hi,

Yea, wildcard text does not get analyzed when used in a query string.
This is for the simple reason that analysis might produce more than one
token and then it can be confusing as to where to apply the wildcard
element.

Having said that, I think I saw somewhere a query parser that did
analysis on wildcard. I need to find it. Maybe it can be added as an option.

-shay.banon

On Wednesday, March 16, 2011 at 5:59 PM, Enrique Medina Montenegro wrote:

And just to make it more clear, the bug would be ONLY when using
wildcards...

But I'm afraid this is a Lucene bug, but Shay can comment on it in the
issue...

In the meantime, a workaround is to analyze the search words manually
before composing the query, which is easy to implement but adds some hassle
to the process, obviously.

On Wed, Mar 16, 2011 at 4:56 PM, Clinton Gormley <clinton@iannounce.co.uk

wrote:

On Wed, 2011-03-16 at 16:55 +0100, Enrique Medina Montenegro wrote:

The more I think about this the more I get convinced that the issue is
with analyzers using stemmers, and the fact that the search ignores
the stemmer when parsing the query terms...

Exactly. That's why I said there is a bug.

clint

Enrique_Medina_Monte · March 17, 2011, 9:59am

Shay,

I have filed an enhancement for your consideration to allow pre-analysis of
search terms in a wildcard 'query_string' query:

github.com/elastic/elasticsearch

Query: Provide an option to analyze wildcard/prefix in query_string / field queries

opened 09:54AM - 17 Mar 11 UTC

closed 08:01PM - 17 Mar 11 UTC

emedina

>enhancement v0.16.0

Add a flag called `analyze_wildcard` to both `query_string` and `field` queries,… once set, a best effort will be made to analyze wildcard and prefix queries as well. More details: When we use an analyzer that stems terms into tokens, and then later we want to search against those analyzed terms using a wildcard, by default the search terms are not analyzed, as that analysis could lead into several tokens and the search engine would not be sure which one to use: http://www.jguru.com/faq/view.jsp?EID=538312 However, in certain circumstances, when the liability of the search can be somehow constrained in favor of better expected results, it would be nice to tell the search engine to analyze the wildcard terms before executing the search, therefore allowing for a more precise search (at least, expected). Let's put here an example with the Spanish analyzer (which uses the snowball stemmer): - We index the phrase "I have an iPhone" - We index the phrase "I love the triad iPad/iPhone/iPod" - We index the phrase "I found the perfect combination: iPhone/MP3" If we use the standard current 'query_string', when searching for "*phone*", we will only get the last phrase, due to the way in which the terms have been analyzed: "I have an iPhone": {"tokens":[{"token":"i","start_offset":0,"end_offset":1,"type":"<ALPHANUM>","position":1},{"token":"hav","start_offset":2,"end_offset":6,"type":"<ALPHANUM>","position":2},{"token":"an","start_offset":7,"end_offset":9,"type":"<ALPHANUM>","position":3},{"token":"iphon","start_offset":10,"end_offset":16,"type":"<ALPHANUM>","position":4}]} "I love the triad iPad/iPhone/iPod": {"tokens":[{"token":"i","start_offset":0,"end_offset":1,"type":"<ALPHANUM>","position":1},{"token":"lov","start_offset":2,"end_offset":6,"type":"<ALPHANUM>","position":2},{"token":"the","start_offset":7,"end_offset":10,"type":"<ALPHANUM>","position":3},{"token":"tri","start_offset":11,"end_offset":16,"type":"<ALPHANUM>","position":4},{"token":"ipad","start_offset":17,"end_offset":21,"type":"<ALPHANUM>","position":5},{"token":"iphon","start_offset":22,"end_offset":28,"type":"<ALPHANUM>","position":6},{"token":"ipod","start_offset":29,"end_offset":33,"type":"<ALPHANUM>","position":7}]} "I found the perfect combination: iPhone/MP3": {"tokens":[{"token":"i","start_offset":0,"end_offset":1,"type":"<ALPHANUM>","position":1},{"token":"found","start_offset":2,"end_offset":7,"type":"<ALPHANUM>","position":2},{"token":"the","start_offset":8,"end_offset":11,"type":"<ALPHANUM>","position":3},{"token":"perfect","start_offset":12,"end_offset":19,"type":"<ALPHANUM>","position":4},{"token":"combination","start_offset":20,"end_offset":31,"type":"<ALPHANUM>","position":5},{"token":"iphone/mp3","start_offset":33,"end_offset":43,"type":"<NUM>","position":6}]} See how the latter stems "iPhone/MP3" as "iphone/mp3"? Hence this is the only one matching a 'query_string' equal to "*phone*" (and similar 'unexpected' results occur when using just one leading or trailing wildcard as well). This result would be dissapointing for the user, as she'd expect at least something like "iPhone" or even "telephone" to be returned as a result, but due to fact that the Spanish analyzer will always remove the trailing 'e' from most of the words, it won't find them. So enhancement would be to provide a mechanism, in the form of a parameter, for instance, in the 'query_string', that would tell the ES query parser to analyze those search terms surrounded by wildcards (i.e. either enclosed completely, or just with a leading or trailing wildcard). Following our previous example, a 'query_string' for "*phone*" would be actually analyzed in the Spanish analyzer as "*phon*" therefore returning absolutely all the phrases previously created, which would be the expected and reasonable behaviour from a user's perspective. Of course, it could have some side effects on other searches, but as a parameter, it would be up to the search designer to either use it or not.

Not sure how to label it as enhancement, so left it as issue so far.

Looking forward to hearing your feedback on it.

Regards.

On Thu, Mar 17, 2011 at 10:06 AM, Enrique Medina Montenegro <
e.medina.m@gmail.com> wrote:

As commented below in this thread, I eventually went for the workaround of
analyzing the search terms before enclosing them between wildcards and
executing the 'query_string'. As my analyzer is a simple spanish snowball, I
guess I shouldn't get more than 1 token per word, but in any case, my code
would only pick up the first one.

So my question is: would this be a good feature to include in ES? Basically
there could be a property/parameter in the 'query_string' (and even in the
'wildcard' query) that would tell ES to analyze the search terms before
querying when they are enclosed between wildcards (or just have left or
right wildcard). By default it would be set to false, so regular behaviour
applies.

On Wed, Mar 16, 2011 at 8:46 PM, Shay Banon shay.banon@elasticsearch.comwrote:

It really depends on the analyzer you use, and because its unknown on
the query parser level, I guess it is not supported. There are analyzers
that can result in multiple tokens, like ngram ones.

On Wednesday, March 16, 2011 at 7:51 PM, Enrique Medina Montenegro wrote:

Shay,

What would be an example of a word producing more than one token?

Thanks.

On Wed, Mar 16, 2011 at 6:40 PM, Shay Banon <shay.banon@elasticsearch.com

wrote:

Hi,

Yea, wildcard text does not get analyzed when used in a query string.
This is for the simple reason that analysis might produce more than one
token and then it can be confusing as to where to apply the wildcard
element.

Having said that, I think I saw somewhere a query parser that did
analysis on wildcard. I need to find it. Maybe it can be added as an option.

-shay.banon

On Wednesday, March 16, 2011 at 5:59 PM, Enrique Medina Montenegro wrote:

And just to make it more clear, the bug would be ONLY when using
wildcards...

But I'm afraid this is a Lucene bug, but Shay can comment on it in the
issue...

In the meantime, a workaround is to analyze the search words manually
before composing the query, which is easy to implement but adds some hassle
to the process, obviously.

On Wed, Mar 16, 2011 at 4:56 PM, Clinton Gormley <
clinton@iannounce.co.uk> wrote:

On Wed, 2011-03-16 at 16:55 +0100, Enrique Medina Montenegro wrote:

The more I think about this the more I get convinced that the issue is
with analyzers using stemmers, and the fact that the search ignores
the stemmer when parsing the query terms...

Exactly. That's why I said there is a bug.

clint

kimchy · March 17, 2011, 10:13am

Got it. Certainly possible.
On Thursday, March 17, 2011 at 11:59 AM, Enrique Medina Montenegro wrote:

Shay,

I have filed an enhancement for your consideration to allow pre-analysis of search terms in a wildcard 'query_string' query:

Query: Provide an option to analyze wildcard/prefix in query_string / field queries · Issue #787 · elastic/elasticsearch · GitHub

Not sure how to label it as enhancement, so left it as issue so far.

Looking forward to hearing your feedback on it.

Regards.

On Thu, Mar 17, 2011 at 10:06 AM, Enrique Medina Montenegro e.medina.m@gmail.com wrote:

As commented below in this thread, I eventually went for the workaround of analyzing the search terms before enclosing them between wildcards and executing the 'query_string'. As my analyzer is a simple spanish snowball, I guess I shouldn't get more than 1 token per word, but in any case, my code would only pick up the first one.

So my question is: would this be a good feature to include in ES? Basically there could be a property/parameter in the 'query_string' (and even in the 'wildcard' query) that would tell ES to analyze the search terms before querying when they are enclosed between wildcards (or just have left or right wildcard). By default it would be set to false, so regular behaviour applies.

On Wed, Mar 16, 2011 at 8:46 PM, Shay Banon shay.banon@elasticsearch.com wrote:

It really depends on the analyzer you use, and because its unknown on the query parser level, I guess it is not supported. There are analyzers that can result in multiple tokens, like ngram ones.
On Wednesday, March 16, 2011 at 7:51 PM, Enrique Medina Montenegro wrote:

Shay,

What would be an example of a word producing more than one token?

Thanks.

On Wed, Mar 16, 2011 at 6:40 PM, Shay Banon shay.banon@elasticsearch.com wrote:

Hi,

Yea, wildcard text does not get analyzed when used in a query string. This is for the simple reason that analysis might produce more than one token and then it can be confusing as to where to apply the wildcard element.

Having said that, I think I saw somewhere a query parser that did analysis on wildcard. I need to find it. Maybe it can be added as an option.

-shay.banon
On Wednesday, March 16, 2011 at 5:59 PM, Enrique Medina Montenegro wrote:

And just to make it more clear, the bug would be ONLY when using wildcards...

But I'm afraid this is a Lucene bug, but Shay can comment on it in the issue...

In the meantime, a workaround is to analyze the search words manually before composing the query, which is easy to implement but adds some hassle to the process, obviously.

On Wed, Mar 16, 2011 at 4:56 PM, Clinton Gormley clinton@iannounce.co.uk wrote:

On Wed, 2011-03-16 at 16:55 +0100, Enrique Medina Montenegro wrote:

The more I think about this the more I get convinced that the issue is
with analyzers using stemmers, and the fact that the search ignores
the stemmer when parsing the query terms...

Exactly. That's why I said there is a bug.

clint