Question on script scoring

Amit_Soni · March 3, 2014, 7:51pm

Hi everyone - Since we were recommended to move away from using "_boost" in
the document, we have been trying to switch over to using custom scoring
(script_score).

We have a float value called boostValue (of type float) in the document we
are using the following script to boost the score of matched document.
However there seems to be no change in the generated score.

Any idea what might be wrong with this script?

"script_score": {
"script": "doc['boostValue'].empty || (doc['boostValue'].value <=
0) ? _score : (doc['boostValue'].value * _score)"
}

-Amit.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAAOGaQ%2Bz3xBH2CkDeWNyc1Vtcku03VQPo6SjKTpG5dFwJtqTpA%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.

Binh_Ly_2 · March 3, 2014, 9:03pm

Are you using the function_score query in 1.0? If so, can you show your
full query? Thanks!

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/7452f652-f9b0-424c-a3ec-3e991afbd42f%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Clinton_Gormley · March 3, 2014, 9:08pm

It could be that the precedence of || ? : is not what you expect. I've run
into issues with that in mvel before. Try using parentheses to make the
script unambiguous

clint

On 3 March 2014 20:51, Amit Soni amitsoni29@gmail.com wrote:

Hi everyone - Since we were recommended to move away from using "_boost"
in the document, we have been trying to switch over to using custom scoring
(script_score).

We have a float value called boostValue (of type float) in the document we
are using the following script to boost the score of matched document.
However there seems to be no change in the generated score.

Any idea what might be wrong with this script?

"script_score": {
"script": "doc['boostValue'].empty || (doc['boostValue'].value <=
0) ? _score : (doc['boostValue'].value * _score)"
}

-Amit.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAAOGaQ%2Bz3xBH2CkDeWNyc1Vtcku03VQPo6SjKTpG5dFwJtqTpA%40mail.gmail.com
.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAPt3XKQwi5rgu-oy3P3k4QanEr33N4GYYxwRPf-2bCYNXb46Vw%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.

nik9000 · March 3, 2014, 9:14pm

First, .empty is super slow right now:

github.com/elastic/elasticsearch

In MVEL .empty can be way way way slower then .isEmpty()

opened 03:50PM - 11 Feb 14 UTC

closed 11:36AM - 29 Dec 14 UTC

nik9000

This happens when you use `.empty` to check if a field is empty and it actually …is empty. I've made a gist to reproduce it: https://gist.github.com/nik9000/8937202 High level: 1. Hit ~1 million documents with a script score. 2. Do something like `(doc['foo'].empty ? 0 : doc['foo'].value) * doc['bar']`. The `.empty` is the key here. 3. If most of the documents don't have a `foo` then this is really slow. Like, two seconds slow. 4. Instead, switch to this `(doc['foo'].isEmpty() ? 0 : doc['foo'].value) * doc['bar']`. That is faster. .36 seconds or so. Not super speedy, but much better. I'm not completely sure what is happening but my guess is that the MVEL optimizer is optimizing .empty is a call to .isEmpty on an instance of either ScriptDocValues or ScriptDocValues.Empty. When it hits the wrong one it recovers by catching an IllegalArgumentException thrown by the JVM and then does some munging. The act of filling in the stack trace for that IllegalArgumentException is really really really slow. So slow, I see it in the hot threads: https://gist.github.com/nik9000/8937335 . Workaround: use `.isEmpty()` instead of `.empty`. I'm not sure what the fix ought to be.

Second, be paranoid about order of operations with MVEL. I'd play around
with parrens until it worked.
Third, multiply is the default boost mode so you can just leave out the
_score which might make the MVEL simpler.

On Mon, Mar 3, 2014 at 4:03 PM, Binh Ly binhly_es@yahoo.com wrote:

Are you using the function_score query in 1.0? If so, can you show your
full query? Thanks!

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/7452f652-f9b0-424c-a3ec-3e991afbd42f%40googlegroups.com
.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAPmjWd3tgEtVD4Lt6Ce3i225M2ZHq%2BuBDc1Zsrh3xH6fKYtSLw%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.

Amit_Soni · March 4, 2014, 2:56am

Thanks much Nikolas and Clint.

@Binh - Below is my complete query:

{
"from": 0,
"size": 15,
"query": {
"function_score": {
"query": {
"filtered": {
"query": {
"simple_query_string": {
"query": "some query",
"fields": [
"searchKeywords",
"website",
"name^2.0"
],
"default_operator": "and"
}
},
"filter": {
"term": {
"region": "some region"
}
}
}
},
"script_score": {
"script": "doc['boostValue'].empty || (doc['boostValue'].value <=
0) ? _score : doc['boostValue'].value * _score"
}
}
},
"explain": false
}

-Amit.

On Mon, Mar 3, 2014 at 1:14 PM, Nikolas Everett nik9000@gmail.com wrote:

First, .empty is super slow right now:
In MVEL .empty can be way way way slower then .isEmpty() · Issue #5086 · elastic/elasticsearch · GitHub
Second, be paranoid about order of operations with MVEL. I'd play around
with parrens until it worked.
Third, multiply is the default boost mode so you can just leave out the
_score which might make the MVEL simpler.

On Mon, Mar 3, 2014 at 4:03 PM, Binh Ly binhly_es@yahoo.com wrote:

Are you using the function_score query in 1.0? If so, can you show your
full query? Thanks!

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/7452f652-f9b0-424c-a3ec-3e991afbd42f%40googlegroups.com
.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAPmjWd3tgEtVD4Lt6Ce3i225M2ZHq%2BuBDc1Zsrh3xH6fKYtSLw%40mail.gmail.com
.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAAOGaQ%2BzgC%3D_mLbRVu%3DDfS1B7VPoRmcKufm8EH7Hv-m93zcMXw%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.

Amit_Soni · March 4, 2014, 3:13am

and yes I am running 1.0.0.

@Nikolas and Clint - I tried your suggestions and played with parenthesis
but didnt have any luck. anything else I am missing?

thanks much for your help!

-Amit.

On Mon, Mar 3, 2014 at 6:56 PM, Amit Soni amitsoni29@gmail.com wrote:

Thanks much Nikolas and Clint.

@Binh - Below is my complete query:

{
"from": 0,
"size": 15,
"query": {
"function_score": {
"query": {
"filtered": {
"query": {
"simple_query_string": {
"query": "some query",
"fields": [
"searchKeywords",
"website",
"name^2.0"
],
"default_operator": "and"
}
},
"filter": {
"term": {
"region": "some region"
}
}
}
},
"script_score": {
"script": "doc['boostValue'].empty || (doc['boostValue'].value <=
0) ? _score : doc['boostValue'].value * _score"
}
}
},
"explain": false
}

-Amit.

On Mon, Mar 3, 2014 at 1:14 PM, Nikolas Everett nik9000@gmail.com wrote:

First, .empty is super slow right now:
In MVEL .empty can be way way way slower then .isEmpty() · Issue #5086 · elastic/elasticsearch · GitHub
Second, be paranoid about order of operations with MVEL. I'd play around
with parrens until it worked.
Third, multiply is the default boost mode so you can just leave out the
_score which might make the MVEL simpler.

On Mon, Mar 3, 2014 at 4:03 PM, Binh Ly binhly_es@yahoo.com wrote:

Are you using the function_score query in 1.0? If so, can you show your
full query? Thanks!

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/7452f652-f9b0-424c-a3ec-3e991afbd42f%40googlegroups.com
.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAPmjWd3tgEtVD4Lt6Ce3i225M2ZHq%2BuBDc1Zsrh3xH6fKYtSLw%40mail.gmail.com
.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAAOGaQL1kAXmyo5g%3DcC1x%3DyL789SAdYTfJD9FMOeG0yWD%2BCs2w%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.

Amit_Soni · March 6, 2014, 5:36pm

hello everyone - first of all my apology for following up on this soon
enough.

wondering if anyone has got things working with script scoring and can
share their script, or help me with what might be wrong with the below
script.

"script_score": {
"script": "doc['boostValue'].empty || (doc['boostValue'].value <=
0) ? _score : doc['boostValue'].value * _score"
}

-Amit.

On Mon, Mar 3, 2014 at 7:13 PM, Amit Soni amitsoni29@gmail.com wrote:

and yes I am running 1.0.0.

@Nikolas and Clint - I tried your suggestions and played with parenthesis
but didnt have any luck. anything else I am missing?

thanks much for your help!

-Amit.

On Mon, Mar 3, 2014 at 6:56 PM, Amit Soni amitsoni29@gmail.com wrote:

Thanks much Nikolas and Clint.

@Binh - Below is my complete query:

{
"from": 0,
"size": 15,
"query": {
"function_score": {
"query": {
"filtered": {
"query": {
"simple_query_string": {
"query": "some query",
"fields": [
"searchKeywords",
"website",
"name^2.0"
],
"default_operator": "and"
}
},
"filter": {
"term": {
"region": "some region"
}
}
}
},
"script_score": {
"script": "doc['boostValue'].empty || (doc['boostValue'].value
<= 0) ? _score : doc['boostValue'].value * _score"
}
}
},
"explain": false
}

-Amit.

On Mon, Mar 3, 2014 at 1:14 PM, Nikolas Everett nik9000@gmail.comwrote:

First, .empty is super slow right now:
In MVEL .empty can be way way way slower then .isEmpty() · Issue #5086 · elastic/elasticsearch · GitHub
Second, be paranoid about order of operations with MVEL. I'd play
around with parrens until it worked.
Third, multiply is the default boost mode so you can just leave out the
_score which might make the MVEL simpler.

On Mon, Mar 3, 2014 at 4:03 PM, Binh Ly binhly_es@yahoo.com wrote:

Are you using the function_score query in 1.0? If so, can you show your
full query? Thanks!

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/7452f652-f9b0-424c-a3ec-3e991afbd42f%40googlegroups.com
.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAPmjWd3tgEtVD4Lt6Ce3i225M2ZHq%2BuBDc1Zsrh3xH6fKYtSLw%40mail.gmail.com
.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAAOGaQL5P3u3Qx9OX7kQEdT1DRxvkNCow8XgLYdVeeB8c0F0bA%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.

Amit_Soni · March 14, 2014, 5:31am

Hi all - After a good amount of debugging, I figured out the problem and
wanted to share with this group. I had defined the mapping for 'boostValue'
wherein I had "index=no", my bad

After I changed mapping to "index": "not_analyzed" for this boost field, it
worked fine.

-Amit.

On Thu, Mar 6, 2014 at 9:36 AM, Amit Soni amitsoni29@gmail.com wrote:

hello everyone - first of all my apology for following up on this soon
enough.

wondering if anyone has got things working with script scoring and can
share their script, or help me with what might be wrong with the below
script.

"script_score": {
"script": "doc['boostValue'].empty || (doc['boostValue'].value <=
0) ? _score : doc['boostValue'].value * _score"
}

-Amit.

On Mon, Mar 3, 2014 at 7:13 PM, Amit Soni amitsoni29@gmail.com wrote:

and yes I am running 1.0.0.

@Nikolas and Clint - I tried your suggestions and played with parenthesis
but didnt have any luck. anything else I am missing?

thanks much for your help!

-Amit.

On Mon, Mar 3, 2014 at 6:56 PM, Amit Soni amitsoni29@gmail.com wrote:

Thanks much Nikolas and Clint.

@Binh - Below is my complete query:

{
"from": 0,
"size": 15,
"query": {
"function_score": {
"query": {
"filtered": {
"query": {
"simple_query_string": {
"query": "some query",
"fields": [
"searchKeywords",
"website",
"name^2.0"
],
"default_operator": "and"
}
},
"filter": {
"term": {
"region": "some region"
}
}
}
},
"script_score": {
"script": "doc['boostValue'].empty || (doc['boostValue'].value
<= 0) ? _score : doc['boostValue'].value * _score"
}
}
},
"explain": false
}

-Amit.

On Mon, Mar 3, 2014 at 1:14 PM, Nikolas Everett nik9000@gmail.comwrote:

First, .empty is super slow right now:
In MVEL .empty can be way way way slower then .isEmpty() · Issue #5086 · elastic/elasticsearch · GitHub
Second, be paranoid about order of operations with MVEL. I'd play
around with parrens until it worked.
Third, multiply is the default boost mode so you can just leave out the
_score which might make the MVEL simpler.

On Mon, Mar 3, 2014 at 4:03 PM, Binh Ly binhly_es@yahoo.com wrote:

Are you using the function_score query in 1.0? If so, can you show
your full query? Thanks!

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/7452f652-f9b0-424c-a3ec-3e991afbd42f%40googlegroups.com
.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAPmjWd3tgEtVD4Lt6Ce3i225M2ZHq%2BuBDc1Zsrh3xH6fKYtSLw%40mail.gmail.com
.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAAOGaQJEE4sesCrGuVfKcXqo8qOV6jONr%3D9NRfvnx4N0WSytfg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.