Hi Ivan,
I followed your advice and started using explain api for query_string but I
guess in process I found a bug (don't know if it really is a bug or
intended behaviour of query_string). This is going to be a long post,
please be patient with me.
I'm using a doc:{name:"new delhi to goa",st:"goa"}
On using analyzer api for indexing I got these tokens:
{
"tokens" : [ {
"token" : "new",
"start_offset" : 0,
"end_offset" : 3,
"type" : "word",
"position" : 1
}, {
"token" : "new",
"start_offset" : 0,
"end_offset" : 9,
"type" : "word",
"position" : 1
}, {
"token" : "new ",
"start_offset" : 0,
"end_offset" : 9,
"type" : "word",
"position" : 1
}, {
"token" : "new d",
"start_offset" : 0,
"end_offset" : 9,
"type" : "word",
"position" : 1
}, {
"token" : "new de",
"start_offset" : 0,
"end_offset" : 9,
"type" : "word",
"position" : 1
}, {
"token" : "new del",
"start_offset" : 0,
"end_offset" : 9,
"type" : "word",
"position" : 1
}, {
"token" : "new delh",
"start_offset" : 0,
"end_offset" : 9,
"type" : "word",
"position" : 1
}, {
"token" : "new delhi",
"start_offset" : 0,
"end_offset" : 9,
"type" : "word",
"position" : 1
}, {
"token" : "new",
"start_offset" : 0,
"end_offset" : 12,
"type" : "word",
"position" : 1
}, {
"token" : "new ",
"start_offset" : 0,
"end_offset" : 12,
"type" : "word",
"position" : 1
}, {
"token" : "new d",
"start_offset" : 0,
"end_offset" : 12,
"type" : "word",
"position" : 1
}, {
"token" : "new de",
"start_offset" : 0,
"end_offset" : 12,
"type" : "word",
"position" : 1
}, {
"token" : "new del",
"start_offset" : 0,
"end_offset" : 12,
"type" : "word",
"position" : 1
}, {
"token" : "new delh",
"start_offset" : 0,
"end_offset" : 12,
"type" : "word",
"position" : 1
}, {
"token" : "new delhi",
"start_offset" : 0,
"end_offset" : 12,
"type" : "word",
"position" : 1
}, {
"token" : "new delhi ",
"start_offset" : 0,
"end_offset" : 12,
"type" : "word",
"position" : 1
}, {
"token" : "new delhi t",
"start_offset" : 0,
"end_offset" : 12,
"type" : "word",
"position" : 1
}, {
"token" : "new delhi to",
"start_offset" : 0,
"end_offset" : 12,
"type" : "word",
"position" : 1
}, {
"token" : "new",
"start_offset" : 0,
"end_offset" : 16,
"type" : "word",
"position" : 1
}, {
"token" : "new ",
"start_offset" : 0,
"end_offset" : 16,
"type" : "word",
"position" : 1
}, {
"token" : "new d",
"start_offset" : 0,
"end_offset" : 16,
"type" : "word",
"position" : 1
}, {
"token" : "new de",
"start_offset" : 0,
"end_offset" : 16,
"type" : "word",
"position" : 1
}, {
"token" : "new del",
"start_offset" : 0,
"end_offset" : 16,
"type" : "word",
"position" : 1
}, {
"token" : "new delh",
"start_offset" : 0,
"end_offset" : 16,
"type" : "word",
"position" : 1
}, {
"token" : "new delhi",
"start_offset" : 0,
"end_offset" : 16,
"type" : "word",
"position" : 1
}, {
"token" : "new delhi ",
"start_offset" : 0,
"end_offset" : 16,
"type" : "word",
"position" : 1
}, {
"token" : "new delhi t",
"start_offset" : 0,
"end_offset" : 16,
"type" : "word",
"position" : 1
}, {
"token" : "new delhi to",
"start_offset" : 0,
"end_offset" : 16,
"type" : "word",
"position" : 1
}, {
"token" : "new delhi to ",
"start_offset" : 0,
"end_offset" : 16,
"type" : "word",
"position" : 1
}, {
"token" : "new delhi to g",
"start_offset" : 0,
"end_offset" : 16,
"type" : "word",
"position" : 1
}, {
"token" : "new delhi to go",
"start_offset" : 0,
"end_offset" : 16,
"type" : "word",
"position" : 1
}, {
"token" : "new delhi to goa",
"start_offset" : 0,
"end_offset" : 16,
"type" : "word",
"position" : 1
}, {
"token" : "del",
"start_offset" : 4,
"end_offset" : 9,
"type" : "word",
"position" : 2
}, {
"token" : "delh",
"start_offset" : 4,
"end_offset" : 9,
"type" : "word",
"position" : 2
}, {
"token" : "delhi",
"start_offset" : 4,
"end_offset" : 9,
"type" : "word",
"position" : 2
}, {
"token" : "del",
"start_offset" : 4,
"end_offset" : 12,
"type" : "word",
"position" : 2
}, {
"token" : "delh",
"start_offset" : 4,
"end_offset" : 12,
"type" : "word",
"position" : 2
}, {
"token" : "delhi",
"start_offset" : 4,
"end_offset" : 12,
"type" : "word",
"position" : 2
}, {
"token" : "delhi ",
"start_offset" : 4,
"end_offset" : 12,
"type" : "word",
"position" : 2
}, {
"token" : "delhi t",
"start_offset" : 4,
"end_offset" : 12,
"type" : "word",
"position" : 2
}, {
"token" : "delhi to",
"start_offset" : 4,
"end_offset" : 12,
"type" : "word",
"position" : 2
}, {
"token" : "del",
"start_offset" : 4,
"end_offset" : 16,
"type" : "word",
"position" : 2
}, {
"token" : "delh",
"start_offset" : 4,
"end_offset" : 16,
"type" : "word",
"position" : 2
}, {
"token" : "delhi",
"start_offset" : 4,
"end_offset" : 16,
"type" : "word",
"position" : 2
}, {
"token" : "delhi ",
"start_offset" : 4,
"end_offset" : 16,
"type" : "word",
"position" : 2
}, {
"token" : "delhi t",
"start_offset" : 4,
"end_offset" : 16,
"type" : "word",
"position" : 2
}, {
"token" : "delhi to",
"start_offset" : 4,
"end_offset" : 16,
"type" : "word",
"position" : 2
}, {
"token" : "delhi to ",
"start_offset" : 4,
"end_offset" : 16,
"type" : "word",
"position" : 2
}, {
"token" : "delhi to g",
"start_offset" : 4,
"end_offset" : 16,
"type" : "word",
"position" : 2
}, {
"token" : "delhi to go",
"start_offset" : 4,
"end_offset" : 16,
"type" : "word",
"position" : 2
}, {
"token" : "delhi to goa",
"start_offset" : 4,
"end_offset" : 16,
"type" : "word",
"position" : 2
}, {
"token" : "to ",
"start_offset" : 10,
"end_offset" : 16,
"type" : "word",
"position" : 3
}, {
"token" : "to g",
"start_offset" : 10,
"end_offset" : 16,
"type" : "word",
"position" : 3
}, {
"token" : "to go",
"start_offset" : 10,
"end_offset" : 16,
"type" : "word",
"position" : 3
}, {
"token" : "to goa",
"start_offset" : 10,
"end_offset" : 16,
"type" : "word",
"position" : 3
}, {
"token" : "goa",
"start_offset" : 13,
"end_offset" : 16,
"type" : "word",
"position" : 4
} ]
}
Now, if I query like: "delhi to goa", I got this by search_analyzer:
{
"tokens" : [ {
"token" : "del",
"start_offset" : 0,
"end_offset" : 5,
"type" : "word",
"position" : 1
}, {
"token" : "delh",
"start_offset" : 0,
"end_offset" : 5,
"type" : "word",
"position" : 1
}, {
"token" : "delhi",
"start_offset" : 0,
"end_offset" : 5,
"type" : "word",
"position" : 1
}, {
"token" : "del",
"start_offset" : 0,
"end_offset" : 8,
"type" : "word",
"position" : 1
}, {
"token" : "delh",
"start_offset" : 0,
"end_offset" : 8,
"type" : "word",
"position" : 1
}, {
"token" : "delhi",
"start_offset" : 0,
"end_offset" : 8,
"type" : "word",
"position" : 1
}, {
"token" : "delhi ",
"start_offset" : 0,
"end_offset" : 8,
"type" : "word",
"position" : 1
}, {
"token" : "delhi t",
"start_offset" : 0,
"end_offset" : 8,
"type" : "word",
"position" : 1
}, {
"token" : "delhi to",
"start_offset" : 0,
"end_offset" : 8,
"type" : "word",
"position" : 1
}, {
"token" : "del",
"start_offset" : 0,
"end_offset" : 12,
"type" : "word",
"position" : 1
}, {
"token" : "delh",
"start_offset" : 0,
"end_offset" : 12,
"type" : "word",
"position" : 1
}, {
"token" : "delhi",
"start_offset" : 0,
"end_offset" : 12,
"type" : "word",
"position" : 1
}, {
"token" : "delhi ",
"start_offset" : 0,
"end_offset" : 12,
"type" : "word",
"position" : 1
}, {
"token" : "delhi t",
"start_offset" : 0,
"end_offset" : 12,
"type" : "word",
"position" : 1
}, {
"token" : "delhi to",
"start_offset" : 0,
"end_offset" : 12,
"type" : "word",
"position" : 1
}, {
"token" : "delhi to ",
"start_offset" : 0,
"end_offset" : 12,
"type" : "word",
"position" : 1
}, {
"token" : "delhi to g",
"start_offset" : 0,
"end_offset" : 12,
"type" : "word",
"position" : 1
}, {
"token" : "delhi to go",
"start_offset" : 0,
"end_offset" : 12,
"type" : "word",
"position" : 1
}, {
"token" : "delhi to goa",
"start_offset" : 0,
"end_offset" : 12,
"type" : "word",
"position" : 1
}, {
"token" : "to ",
"start_offset" : 6,
"end_offset" : 12,
"type" : "word",
"position" : 2
}, {
"token" : "to g",
"start_offset" : 6,
"end_offset" : 12,
"type" : "word",
"position" : 2
}, {
"token" : "to go",
"start_offset" : 6,
"end_offset" : 12,
"type" : "word",
"position" : 2
}, {
"token" : "to goa",
"start_offset" : 6,
"end_offset" : 12,
"type" : "word",
"position" : 2
}, {
"token" : "goa",
"start_offset" : 9,
"end_offset" : 12,
"type" : "word",
"position" : 3
} ]
}
On using explain api, it gives me following:
{text=new delhi to goa,boostFactor=9.820192307,po=9.82}
510.39673 = custom score, product of:
510.39673 = script score function: composed of:
510.39673 = sum of:
371.12375 = max of:
371.12375 = sum of:
104.61707 = weight(text:del in 1003990)
[PerFieldSimilarity], result of:
104.61707 = score(doc=1003990,freq=5.0 = termFreq=5.0
), product of:
0.43576795 = queryWeight, product of:
5.368244 = idf(docFreq=53067, maxDocs=4187328)
0.08117513 = queryNorm
240.0752 = fieldWeight in 1003990, product of:
2.236068 = tf(freq=5.0), with freq of:
5.0 = termFreq=5.0
5.368244 = idf(docFreq=53067, maxDocs=4187328)
20.0 = fieldNorm(doc=1003990)
133.24011 = weight(text:delh in 1003990)
[PerFieldSimilarity], result of:
133.24011 = score(doc=1003990,freq=5.0 = termFreq=5.0
), product of:
0.49178073 = queryWeight, product of:
6.058268 = idf(docFreq=26616, maxDocs=4187328)
0.08117513 = queryNorm
270.934 = fieldWeight in 1003990, product of:
2.236068 = tf(freq=5.0), with freq of:
5.0 = termFreq=5.0
6.058268 = idf(docFreq=26616, maxDocs=4187328)
20.0 = fieldNorm(doc=1003990)
133.26657 = weight(text:delhi in 1003990)
[PerFieldSimilarity], result of:
133.26657 = score(doc=1003990,freq=5.0 = termFreq=5.0
), product of:
0.49182954 = queryWeight, product of:
6.0588694 = idf(docFreq=26600, maxDocs=4187328)
0.08117513 = queryNorm
270.96088 = fieldWeight in 1003990, product of:
2.236068 = tf(freq=5.0), with freq of:
5.0 = termFreq=5.0
6.0588694 = idf(docFreq=26600, maxDocs=4187328)
20.0 = fieldNorm(doc=1003990)
139.27298 = max of:
139.27298 = weight(text:goa^20.0 in 1003990)
[PerFieldSimilarity], result of:
139.27298 = score(doc=1003990,freq=3.0 = termFreq=3.0
), product of:
0.5712808 = queryWeight, product of:
20.0 = boost
7.037633 = idf(docFreq=9995, maxDocs=4187328)
0.004058757 = queryNorm
243.79076 = fieldWeight in 1003990, product of:
1.7320508 = tf(freq=3.0), with freq of:
3.0 = termFreq=3.0
7.037633 = idf(docFreq=9995, maxDocs=4187328)
20.0 = fieldNorm(doc=1003990)
1.0 = queryBoost
Though the above explain shows the results for:
del
delh
delhi
goa
But not getting results for other tokens which were generated by my search
analyzer. Why is it so ?
I have read that query_string uses query parser which is based on Lucene by
default. So, My guess is query_string is using a whitespace tokenizer after
my tokens are generated by search analyzer, am I correct ? How can I make
query_string to calculate score for all the tokens which are generated by
search_analyzer. Please correct me if I am wrong.
There is one more things which I noticed,
I'm using a query time boost on one of my doc field but it is not working
the way I thought it would work. In the above explain you can see, there is
a boost associated with goa but not with delhi, though but goa and delhi
are present in original doc. My guess for this is,
query_string applies boost to only terms where a term is a token of a user
typed string which is not analyzed by any analyzer because in the above
example, goa is kept as it is but delhi is being analyzed. Am I correct ?
Waiting a reply !!!
Thanks
On Tue, Feb 4, 2014 at 1:03 AM, Ivan Brusic ivan@brusic.com wrote:
Try using the plugin I suggested and/or the explain API.
Elasticsearch Platform — Find real-time answers at scale | Elastic
On Mon, Feb 3, 2014 at 11:26 AM, Mukul Gupta mukulnitkkr@gmail.comwrote:
Hi Ivan,
Thanks for the reply. Can you please tell how query_string works
internally ? In the documentation, it says query is parsed internally. I
want to know what is the meaning of this ? Let's say I 'm searching for
"delhi t" so how query_string will query for this string. Also assume I'm
using search_analyzer = standard with filter
lowercase,asciifolding,suggestion_shingle
(min_gram=2,max_gram=2),edgengram(min_gram=2,max_gram=15). I know after
search_analyzer I'll have the following:
{
"tokens" : [ {
"token" : "de",
"start_offset" : 0,
"end_offset" : 5,
"type" : "word",
"position" : 1
}, {
"token" : "del",
"start_offset" : 0,
"end_offset" : 5,
"type" : "word",
"position" : 1
}, {
"token" : "delh",
"start_offset" : 0,
"end_offset" : 5,
"type" : "word",
"position" : 1
}, {
"token" : "delhi",
"start_offset" : 0,
"end_offset" : 5,
"type" : "word",
"position" : 1
}, {
"token" : "de",
"start_offset" : 0,
"end_offset" : 7,
"type" : "word",
"position" : 1
}, {
"token" : "del",
"start_offset" : 0,
"end_offset" : 7,
"type" : "word",
"position" : 1
}, {
"token" : "delh",
"start_offset" : 0,
"end_offset" : 7,
"type" : "word",
"position" : 1
}, {
"token" : "delhi",
"start_offset" : 0,
"end_offset" : 7,
"type" : "word",
"position" : 1
}, {
"token" : "delhi ",
"start_offset" : 0,
"end_offset" : 7,
"type" : "word",
"position" : 1
}, {
"token" : "delhi t",
"start_offset" : 0,
"end_offset" : 7,
"type" : "word",
"position" : 1
} ]
}
So, now how a query string will search in a document ?
Thanks
On Tue, Feb 4, 2014 at 12:39 AM, Ivan Brusic ivan@brusic.com wrote:
The key difference is that a query_string query (with multiple fields
and the AND operator) will match when each term exists in at least one
field while a multi-match query (also using AND) matches only when all the
terms exist in at least one field. The terms of a query_string query do not
need to exist in the same field, while it does matter in a multi-match.
Play around with the Sense Chrome plugin and some sample data:
https://chrome.google.com/webstore/detail/sense/doinijnbnggojdlcjifpdckfokbbfpbo
Cheers,
Ivan
On Mon, Feb 3, 2014 at 10:55 AM, coder mukulnitkkr@gmail.com wrote:
Hi,
Can anyone please tell me a detailed difference between these two
queries. I studied the documentation but not able to figure out the
difference between two. Can anyone please explain it with some examples in
a more detailed fashion. I expect query string to give me docs which
matches maximum number of terms which are generated by search_analyzer to
indexed docs but it is not happening that way.
Please help !!!
Thanks
--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/fbc68c13-bcb9-40cb-9726-becbef14f278%40googlegroups.com
.
For more options, visit https://groups.google.com/groups/opt_out.
--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQBP-3JhfnX0Qoj0uYnBBUX_V9Oysc45tg40eK7b7NB0rQ%40mail.gmail.com
.
For more options, visit https://groups.google.com/groups/opt_out.
--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAAVTvp6XQ%3DJWCN9KkSS4cjs1ua1rtcy3Fcr8xG-AFN4dBC5HNw%40mail.gmail.com
.
For more options, visit https://groups.google.com/groups/opt_out.
--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQBRM_ZF4bvQCdOvQnyxgz%3DNrQAbAG02e8CGTHQoPrDrQw%40mail.gmail.com
.
For more options, visit https://groups.google.com/groups/opt_out.
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAAVTvp4SDcKVOm6nh_-NQg7O5bVeQGLmQ3HD7UvAdu%3DB9m_Nxw%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.