Custom_score requires specific attribute ordering

Elastic Stack Elasticsearch
1

Hi,

I have the following query fragment which works fine in ES 0.13.0:

"query" : {
    "custom_score" : {
        "filtered" : {
            "query" : {
                "bool" : {
                    "must_not" : {
                        "term" : {
                            "group.vip" : "T"
                        }
                    },
                    "must" : {
                        "match_all" : {

                        }
                    }
                }
            },
           "params" : {
                "categoryId" : "77205"
           },
           "lang" : "js",
           "script" : "parseFloat(eval(doc['catSequences'].value)

[categoryId])",
"filter" : {
"term" : {
"category.pathIds" : "77205"
}
}

        }
    }
}

But if the "script" element comes after the "filter" object then the I
get the error:

QueryParsingException[[books] [custom_score] requires 'script' field];

Unfortunately I don't have any control over the order object
attributes are generated ( I don't think that order is significant in
json).

My question is should Elastic Search not care about the order or do I
change my json library?

Thanks

Tim

2

Hi Time

You are not nesting your query correctly.

You have:

"query": {
"custom_score": {
"filtered": {
"query": {...},
"params": {...},
"langs": {...},
"script": {...},
"filter": {...}

It should be:

"query": {
"custom_score": {
"query": {
"filtered": { "filter": {..}, "query": {...}}
},
"params": {...},
"langs": {...},
"script": {...},

See here for more
http://www.elasticsearch.com/docs/elasticsearch/rest_api/query_dsl/custom_score_query

Also, consider using constant_score instead of match_all:
http://www.elasticsearch.com/docs/elasticsearch/rest_api/query_dsl/constant_score_query

clint

3

Thanks for pointing that out. The query is now working fine

"query" : {
    "custom_score" : {
        "query" : {
            "bool" : {
                "must_not" : {
                    "term" : {
                        "group.vip" : "T"
                    }
                },
                "must" : {
                    "constant_score" : {
                        "filter" : {
                            "term" : {
                                "category.pathIds" : "77705"
                            }
                        }
                    }
                }
            }
        },
        "script" : "parseFloat(eval(doc['catSequences'].value)

[categoryId])",
"params" : {
"categoryId" : 77705
},
"lang" : "js"
}

On Nov 30, 11:55 am, Clinton Gormley clin...@iannounce.co.uk wrote:

Hi Time

You are not nesting your query correctly.

You have:

"query": {
"custom_score": {
"filtered": {
"query": {...},
"params": {...},
"langs": {...},
"script": {...},
"filter": {...}

It should be:

"query": {
"custom_score": {
"query": {
"filtered": { "filter": {..}, "query": {...}}
},
"params": {...},
"langs": {...},
"script": {...},

See here for morehttp://www.elasticsearch.com/docs/elasticsearch/rest_api/query_dsl/cu...

Also, consider using constant_score instead of match_all:http://www.elasticsearch.com/docs/elasticsearch/rest_api/query_dsl/co...

clint

4

On Tue, 2010-11-30 at 04:45 -0800, Timber wrote:

Thanks for pointing that out. The query is now working fine

You can reduce that further to this:

"query" : {
"custom_score" : {
"query" : {
"bool" : {
"must_not" : {
"term" : {
"group.vip" : "T"
}
},
"must" : {
"term" : {
"category.pathIds" : "77705"
}
}
}
},
"script" : "parseFloat(eval(doc['catSequences'].value)[categoryId])",
"params" : {
"categoryId" : 77705
},
"lang" : "js"
}
}

clint

5

Hi,

Great that its working. I am not sure I got the script you are executing,
what are you trying to do? (just want to see if it can be optimized).

On Tue, Nov 30, 2010 at 3:08 PM, Clinton Gormley clinton@iannounce.co.ukwrote:

On Tue, 2010-11-30 at 04:45 -0800, Timber wrote:

Thanks for pointing that out. The query is now working fine

You can reduce that further to this:

"query" : {
"custom_score" : {
"query" : {
"bool" : {
"must_not" : {
"term" : {
"group.vip" : "T"
}
},
"must" : {
"term" : {
"category.pathIds" : "77705"
}
}
}
},
"script" :
"parseFloat(eval(doc['catSequences'].value)[categoryId])",
"params" : {
"categoryId" : 77705
},
"lang" : "js"
}
}

clint

6

Hi Shay,

I am trying too allow the products in each of our categories to be
ordered in a category specific order. The first page of two of
products in each category browsed are to be displayed in an order
determined by our marketing team. When a search term is used we just
use relevance.

The catSequences field is indexed as "catSequences":
"({76715:373.0,48124:141.0,76704:62.0,48122:325.0,76710:21.0})" which
puts each product in an ordered position in the category. This is
obviously a sparse mapping.

On a side note I had no luck using groovy as the scripting language.

Thanks

Tim

On Nov 30, 9:25 pm, Shay Banon shay.ba...@elasticsearch.com wrote:

Hi,

Great that its working. I am not sure I got the script you are executing,
what are you trying to do? (just want to see if it can be optimized).

On Tue, Nov 30, 2010 at 3:08 PM, Clinton Gormley clin...@iannounce.co.ukwrote:

On Tue, 2010-11-30 at 04:45 -0800, Timber wrote:

Thanks for pointing that out. The query is now working fine

You can reduce that further to this:

"query" : {
"custom_score" : {
"query" : {
"bool" : {
"must_not" : {
"term" : {
"group.vip" : "T"
}
},
"must" : {
"term" : {
"category.pathIds" : "77705"
}
}
}
},
"script" :
"parseFloat(eval(doc['catSequences'].value)[categoryId])",
"params" : {
"categoryId" : 77705
},
"lang" : "js"
}
}

clint

7

I think I understand, not sure I understand the script though (maybe gmail
garbled it up). What was the problem with groovy?

Also, you can sort on several fields, not just one, maybe that would help as
well?

On Wed, Dec 1, 2010 at 12:33 AM, Timber tim.berkeley@gmail.com wrote:

Hi Shay,

I am trying too allow the products in each of our categories to be
ordered in a category specific order. The first page of two of
products in each category browsed are to be displayed in an order
determined by our marketing team. When a search term is used we just
use relevance.

The catSequences field is indexed as "catSequences":
"({76715:373.0,48124:141.0,76704:62.0,48122:325.0,76710:21.0})" which
puts each product in an ordered position in the category. This is
obviously a sparse mapping.

On a side note I had no luck using groovy as the scripting language.

Thanks

Tim

On Nov 30, 9:25 pm, Shay Banon shay.ba...@elasticsearch.com wrote:

Hi,

Great that its working. I am not sure I got the script you are
executing,
what are you trying to do? (just want to see if it can be optimized).

On Tue, Nov 30, 2010 at 3:08 PM, Clinton Gormley <
clin...@iannounce.co.uk>wrote:

On Tue, 2010-11-30 at 04:45 -0800, Timber wrote:

Thanks for pointing that out. The query is now working fine

You can reduce that further to this:

"query" : {
"custom_score" : {
"query" : {
"bool" : {
"must_not" : {
"term" : {
"group.vip" : "T"
}
},
"must" : {
"term" : {
"category.pathIds" : "77705"
}
}
}
},
"script" :
"parseFloat(eval(doc['catSequences'].value)[categoryId])",
"params" : {
"categoryId" : 77705
},
"lang" : "js"
}
}

clint

8

Hi,

The script parseFloat(eval(doc['catSequences'].value)[categoryId])

takes the value of catSequences for the product:

({76715:373.0,48124:141.0,76704:62.0,48122:325.0,76710:21.0})

and evals it so that we can index the object by the currently selected
categoryId. This gives us a value for the score. 62.0 when categoryId
is set to 76704

Life would be so much simpler if a product only existed in a single
category or the display order was not determined by the category!

Extracting the score value may be faster using string parsing rather
than eval, I have not tested this yet. Still pretty much prototyping.

We already use sorting on multiple fields, once the category entries
are beyond the first few pages there is no position set so we sort the
rest by best selling.

The problem with groovy is I get the error [script_lang not supported
[groovy]] when I set "lang" : "groovy". It looks like the plugin has
installed correctly, I have a lang-groovy directory in plugins which
contains the two jars. Haven't spent too much time looking into this
yet.

On Dec 1, 12:43 am, Shay Banon shay.ba...@elasticsearch.com wrote:

I think I understand, not sure I understand the script though (maybe gmail
garbled it up). What was the problem with groovy?

Also, you can sort on several fields, not just one, maybe that would help as
well?

On Wed, Dec 1, 2010 at 12:33 AM, Timber tim.berke...@gmail.com wrote:

Hi Shay,

I am trying too allow the products in each of our categories to be
ordered in a category specific order. The first page of two of
products in each category browsed are to be displayed in an order
determined by our marketing team. When a search term is used we just
use relevance.

The catSequences field is indexed as "catSequences":
"({76715:373.0,48124:141.0,76704:62.0,48122:325.0,76710:21.0})" which
puts each product in an ordered position in the category. This is
obviously a sparse mapping.

On a side note I had no luck using groovy as the scripting language.

Thanks

Tim

On Nov 30, 9:25 pm, Shay Banon shay.ba...@elasticsearch.com wrote:

Hi,

Great that its working. I am not sure I got the script you are
executing,
what are you trying to do? (just want to see if it can be optimized).

On Tue, Nov 30, 2010 at 3:08 PM, Clinton Gormley <
clin...@iannounce.co.uk>wrote:

On Tue, 2010-11-30 at 04:45 -0800, Timber wrote:

Thanks for pointing that out. The query is now working fine

You can reduce that further to this:

"query" : {
"custom_score" : {
"query" : {
"bool" : {
"must_not" : {
"term" : {
"group.vip" : "T"
}
},
"must" : {
"term" : {
"category.pathIds" : "77705"
}
}
}
},
"script" :
"parseFloat(eval(doc['catSequences'].value)[categoryId])",
"params" : {
"categoryId" : 77705
},
"lang" : "js"
}
}

clint

9

Got it. Strange regarding the groovy part not working, has a test for that.
When you have time, it would be great if you can gist a test case.

On Wed, Dec 1, 2010 at 11:45 AM, Timber tim.berkeley@gmail.com wrote:

Hi,

The script parseFloat(eval(doc['catSequences'].value)[categoryId])

takes the value of catSequences for the product:

({76715:373.0,48124:141.0,76704:62.0,48122:325.0,76710:21.0})

and evals it so that we can index the object by the currently selected
categoryId. This gives us a value for the score. 62.0 when categoryId
is set to 76704

Life would be so much simpler if a product only existed in a single
category or the display order was not determined by the category!

Extracting the score value may be faster using string parsing rather
than eval, I have not tested this yet. Still pretty much prototyping.

We already use sorting on multiple fields, once the category entries
are beyond the first few pages there is no position set so we sort the
rest by best selling.

The problem with groovy is I get the error [script_lang not supported
[groovy]] when I set "lang" : "groovy". It looks like the plugin has
installed correctly, I have a lang-groovy directory in plugins which
contains the two jars. Haven't spent too much time looking into this
yet.

On Dec 1, 12:43 am, Shay Banon shay.ba...@elasticsearch.com wrote:

I think I understand, not sure I understand the script though (maybe
gmail
garbled it up). What was the problem with groovy?

Also, you can sort on several fields, not just one, maybe that would help
as
well?

On Wed, Dec 1, 2010 at 12:33 AM, Timber tim.berke...@gmail.com wrote:

Hi Shay,

I am trying too allow the products in each of our categories to be
ordered in a category specific order. The first page of two of
products in each category browsed are to be displayed in an order
determined by our marketing team. When a search term is used we just
use relevance.

The catSequences field is indexed as "catSequences":
"({76715:373.0,48124:141.0,76704:62.0,48122:325.0,76710:21.0})" which
puts each product in an ordered position in the category. This is
obviously a sparse mapping.

On a side note I had no luck using groovy as the scripting language.

Thanks

Tim

On Nov 30, 9:25 pm, Shay Banon shay.ba...@elasticsearch.com wrote:

Hi,

Great that its working. I am not sure I got the script you are
executing,
what are you trying to do? (just want to see if it can be optimized).

On Tue, Nov 30, 2010 at 3:08 PM, Clinton Gormley <
clin...@iannounce.co.uk>wrote:

On Tue, 2010-11-30 at 04:45 -0800, Timber wrote:

Thanks for pointing that out. The query is now working fine

You can reduce that further to this:

"query" : {
"custom_score" : {
"query" : {
"bool" : {
"must_not" : {
"term" : {
"group.vip" : "T"
}
},
"must" : {
"term" : {
"category.pathIds" : "77705"
}
}
}
},
"script" :
"parseFloat(eval(doc['catSequences'].value)[categoryId])",
"params" : {
"categoryId" : 77705
},
"lang" : "js"
}
}

clint