"Inject" external value for sorting


(Vadim Voituk) #1

Hello,

What i'm trying ot implement is an ES-plugin, that "inject" some kind of
external value (number) into document during search.
Then this value will should be used in a sorting expression ("sort script").

My plugin register a native script, that is used in "script_fields" to
create additional "virtual" field.
But the problem there, that "script_fields" are calculated on a "returnable
top" part of result-set.
So such type of fields can't be used as part of sorting criteria.

Also "injecting" in sort a script (that is OK for my particular case) not
works too, since doc(), source() and fields() return read-only collections.

Another one approach i'm thinking about is - to extend the mvel namespace
with own, custom-made function, that will return this "external" value, and
can be used in an sorting script like:

"script": " doc.score*2 + myCustomFunction(doc._id.value) "

But i can't find the proper way to extend mvel engine for now.

I've found a similar question but there are no solution published there:
https://groups.google.com/forum/?fromgroups#!searchin/elasticsearch/custom$20scoring/elasticsearch/tukeqyZdy7I/W_dZFy0b5ecJ

What is the right way to solve my problem?

Thanks in advance for any ideas, comments and suggestions.


(narinder.izap) #2

You can try with the cusom_filters_score at :
http://www.elasticsearch.org/guide/reference/query-dsl/custom-filters-score-query.html

and you can assign the value to the document based on some specific field
value. I hope its very good solution for your problem, if implemented
properly.

On Monday, 14 May 2012 20:04:37 UTC+5:30, Vadim Voituk wrote:

Hello,

What i'm trying ot implement is an ES-plugin, that "inject" some kind of
external value (number) into document during search.
Then this value will should be used in a sorting expression ("sort
script").

My plugin register a native script, that is used in "script_fields" to
create additional "virtual" field.
But the problem there, that "script_fields" are calculated on a
"returnable top" part of result-set.
So such type of fields can't be used as part of sorting criteria.

Also "injecting" in sort a script (that is OK for my particular case) not
works too, since doc(), source() and fields() return read-only collections.

Another one approach i'm thinking about is - to extend the mvel namespace
with own, custom-made function, that will return this "external" value, and
can be used in an sorting script like:

"script": " doc.score*2 + myCustomFunction(doc._id.value) "

But i can't find the proper way to extend mvel engine for now.

I've found a similar question but there are no solution published there:

https://groups.google.com/forum/?fromgroups#!searchin/elasticsearch/custom$20scoring/elasticsearch/tukeqyZdy7I/W_dZFy0b5ecJ

What is the right way to solve my problem?

Thanks in advance for any ideas, comments and suggestions.


(Vadim Voituk) #3

Hello, thanks for your answer,
but i don't really understand how to add custom calculated field to the
document using custom_filters_scrore.

Could you please describe bit more about this?

On Tuesday, May 15, 2012 6:12:25 AM UTC+2, Narinder Kaur wrote:

You can try with the cusom_filters_score at :

http://www.elasticsearch.org/guide/reference/query-dsl/custom-filters-score-query.html

and you can assign the value to the document based on some specific field
value. I hope its very good solution for your problem, if implemented
properly.

On Monday, 14 May 2012 20:04:37 UTC+5:30, Vadim Voituk wrote:

Hello,

What i'm trying ot implement is an ES-plugin, that "inject" some kind of
external value (number) into document during search.
Then this value will should be used in a sorting expression ("sort
script").

My plugin register a native script, that is used in "script_fields" to
create additional "virtual" field.
But the problem there, that "script_fields" are calculated on a
"returnable top" part of result-set.
So such type of fields can't be used as part of sorting criteria.

Also "injecting" in sort a script (that is OK for my particular case) not
works too, since doc(), source() and fields() return read-only collections.

Another one approach i'm thinking about is - to extend the mvel namespace
with own, custom-made function, that will return this "external" value, and
can be used in an sorting script like:

"script": " doc.score*2 + myCustomFunction(doc._id.value) "

But i can't find the proper way to extend mvel engine for now.

I've found a similar question but there are no solution published there:

https://groups.google.com/forum/?fromgroups#!searchin/elasticsearch/custom$20scoring/elasticsearch/tukeqyZdy7I/W_dZFy0b5ecJ

What is the right way to solve my problem?

Thanks in advance for any ideas, comments and suggestions.


(Klerisson) #4

Just implemented a similar feature, but through a bad coding workaround.

I got the search response in JAVA application, than I performed the custom
score calculation based on the highlighted fields, after that I stored the
documents in a sorted list (PriorityQueue) and, finally, sent the response
to the client.

The overall performance is not good at all, but still enough to my
environment.

Regards,

--
View this message in context: http://elasticsearch-users.115913.n3.nabble.com/Inject-external-value-for-sorting-tp3985874p3998283.html
Sent from the ElasticSearch Users mailing list archive at Nabble.com.


(Shay Banon) #5

Sorry, but I did not understand the question..., script_fields are fields
that only affect the top N hits you return, not the whole doc set. You
probably want to have a script (can be native) that you then use with
custom_score query?

On Mon, May 14, 2012 at 5:34 PM, Vadim Voituk vadim.voituk@gmail.comwrote:

Hello,

What i'm trying ot implement is an ES-plugin, that "inject" some kind of
external value (number) into document during search.
Then this value will should be used in a sorting expression ("sort
script").

My plugin register a native script, that is used in "script_fields" to
create additional "virtual" field.
But the problem there, that "script_fields" are calculated on a
"returnable top" part of result-set.
So such type of fields can't be used as part of sorting criteria.

Also "injecting" in sort a script (that is OK for my particular case) not
works too, since doc(), source() and fields() return read-only collections.

Another one approach i'm thinking about is - to extend the mvel namespace
with own, custom-made function, that will return this "external" value, and
can be used in an sorting script like:

"script": " doc.score*2 + myCustomFunction(doc._id.value) "

But i can't find the proper way to extend mvel engine for now.

I've found a similar question but there are no solution published there:

https://groups.google.com/forum/?fromgroups#!searchin/elasticsearch/custom$20scoring/elasticsearch/tukeqyZdy7I/W_dZFy0b5ecJ

What is the right way to solve my problem?

Thanks in advance for any ideas, comments and suggestions.


(Vadim Voituk) #6

The problem with custom_score is that it overwrites original score value
calculated by ES.
But i want my external "sorting value" to be available in a sorting formula
together with original score and etc.

Something like that
sort: {
_script: {
type: number,
script: doc.score0.34123+ externalScore0.1232
}

Where float numbers are generated on client side (for example depending of
user settings), right before sending query to ES, but the "externalScore" -
it's my external value.

I've implemented this in a way like:

  • I'm receiving the ranking formula from as an parameter to my native
    script
  • Injecting externalScore value to mvel scope
  • Passing it to MvelScriptEngine for further processing

From client-side it looks like:
sort: {
_script: {
language: native,
type: number,
script: my_custom_sorting_script,
params: {
// ... some script specific params
script: doc.score0.34123+ externalScore0.1232 // just a string
param processed by mvel script engine
}
}

It works pretty fast, but i have a strong feeling that there are more
elegant way to do this.
Maybe some kind of plugin endpoint, i've missed?

On Thursday, May 17, 2012 10:29:49 PM UTC+2, kimchy wrote:

Sorry, but I did not understand the question..., script_fields are fields
that only affect the top N hits you return, not the whole doc set. You
probably want to have a script (can be native) that you then use with
custom_score query?

On Mon, May 14, 2012 at 5:34 PM, Vadim Voituk wrote:

Hello,

What i'm trying ot implement is an ES-plugin, that "inject" some kind of
external value (number) into document during search.
Then this value will should be used in a sorting expression ("sort
script").

My plugin register a native script, that is used in "script_fields" to
create additional "virtual" field.
But the problem there, that "script_fields" are calculated on a
"returnable top" part of result-set.
So such type of fields can't be used as part of sorting criteria.

Also "injecting" in sort a script (that is OK for my particular case) not
works too, since doc(), source() and fields() return read-only collections.

Another one approach i'm thinking about is - to extend the mvel namespace
with own, custom-made function, that will return this "external" value, and
can be used in an sorting script like:

"script": " doc.score*2 + myCustomFunction(doc._id.value) "

But i can't find the proper way to extend mvel engine for now.

I've found a similar question but there are no solution published there:

https://groups.google.com/forum/?fromgroups#!searchin/elasticsearch/custom$20scoring/elasticsearch/tukeqyZdy7I/W_dZFy0b5ecJ

What is the right way to solve my problem?

Thanks in advance for any ideas, comments and suggestions.


(Vadim Voituk) #7

As i understand - that's exactly how ES works inside :slight_smile:

On Thursday, May 17, 2012 3:10:55 AM UTC+2, Klérisson Paixão wrote:

Just implemented a similar feature, but through a bad coding workaround.

I got the search response in JAVA application, than I performed the custom
score calculation based on the highlighted fields, after that I stored the
documents in a sorted list (PriorityQueue) and, finally, sent the response
to the client.

The overall performance is not good at all, but still enough to my
environment.

Regards,

--
View this message in context:
http://elasticsearch-users.115913.n3.nabble.com/Inject-external-value-for-sorting-tp3985874p3998283.html
Sent from the ElasticSearch Users mailing list archive at Nabble.com.


(Klerisson) #8

In my case, the score wasn't been calculated, perhaps, because the query type. So, that's why I made that crappy trick. Anyway, I also did not figure out how to handle your situation. :frowning:


(system) #9