Script_fields vs filter script


(oreno) #1

Hi, I notice that using a script_fields that returns true or false values is going much faster then
using the same script but with filter script declaration (so it will filter the docs returning false).

I was sure that the filter script is taking so long because I'm using the source().get(...) method, but turns out that when using the same script, only with script_fields instead, I'm receiving the performance I need. the only problem here is that I want to filter the docs that now have "MessageReverted" = false.

1.Any way I can filter the docs containing "MessageReverted" = false ?(some wrapper query?)
2. Any idea way the filter script is taking much longer then the script field(8000 mill against 250 mill)?

both ways are retrieving the source() for the script logic so it can't be a matter of source fetching as far as I understand.

fast:
...,
"script_fields": {
"MessageReverted": {
"script": "revert",
"lang": "native",
"params": {
"startDate": "2013-05-1",
"endDate": "2013-05-1",
"attributeId": "2365443",
"segmentId": "2365443"
}
}
}

slow:
...,
"filter": {
"script": {
"script": "revert",
"lang": "native",
"params": {
"startDate": "2013-05-1",
"endDate": "2013-05-1",
"attributeId": "2365443",
"segmentId": "2365443"
}
}
}

Any idea?

Thanks in advanced,

Oren


(Alexander Reelsen) #2

Hey,

what kind of query are you executing? Using script fields results in the
scipt only being executed for each search hit, whereas executing it as a
script filter it might need to execute for each document in your index (you
can try to cache the script filter so it might be faster for subsequent
requests).

Hope this helps as a start for optimization, if not, please provide some
more information.

--Alex

On Sun, Jul 7, 2013 at 2:21 PM, oreno oreno@exelate.com wrote:

Hi, I notice that using a script_fields that returns true or false values
is
going much faster then
using the same script but with filter script declaration (so it will filter
the docs returning false).

I was sure that the filter script is taking so long because I'm using the
source().get(...) method, but turns out that when using the same script,
only with script_fields instead, I'm receiving the performance I need. the
only problem here is that I want to filter the docs that now have
"MessageReverted" = false.

1.Any way I can filter the docs containing "MessageReverted" = false
?(some
wrapper query?)
2. Any idea way the filter script is taking much longer then the script
field(8000 mill against 250 mill)?

both ways are retrieving the source() for the script logic so it can't be a
matter of source fetching as far as I understand.

fast:
...,
"script_fields": {
"MessageReverted": {
"script": "revert",
"lang": "native",
"params": {
"startDate": "2013-05-1",
"endDate": "2013-05-1",
"attributeId": "2365443",
"segmentId": "2365443"
}
}
}

slow:
...,
"filter": {
"script": {
"script": "revert",
"lang": "native",
"params": {
"startDate": "2013-05-1",
"endDate": "2013-05-1",
"attributeId": "2365443",
"segmentId": "2365443"
}
}
}

Any idea?

Thanks in advanced,

Oren

--
View this message in context:
http://elasticsearch-users.115913.n3.nabble.com/script-fields-vs-filter-script-tp4037658.html
Sent from the ElasticSearch Users mailing list archive at Nabble.com.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(oreno) #3

Hi Alex,
1.I checked the cache solution but its taking 15 times more then my starting
time (10s against 150s), so that will be a problem since my filter has
dynamic params.
It does go fast once it's stored though. Do you know if it's possible to
do some kind of caching for all source documents for future queries?

2.From what I understand ,both the filter script and the script_field are
suppose to go over each document that results from the prior query.
The only thing I can think of that makes the difference is that the
script_filter actually needs to filter the false documents (for the hit
count) while the script_field only
needs to add the field for the first 10 document returning by default.

I'm trying to figure out how I can speed the response when using source()
on native java script.
I'm assuming the bottle neck is somewhere within creating the response. I
read that using source has some overhead because elasticsearch has to
parse the json source,
but if that was the case here, then I should have received the same big
overhead for both script_field and filter script runs.

All I actually need is the hit count so if I'm correct about the response
parsing and that can be excluded I'll be really glad.

Any idea on the above?

Appreciating your help.

Oren

On Sun, Jul 7, 2013 at 7:13 PM, Alexander Reelsen-2 [via ElasticSearch
Users] ml-node+s115913n4037659h64@n3.nabble.com wrote:

Hey,

what kind of query are you executing? Using script fields results in the
scipt only being executed for each search hit, whereas executing it as a
script filter it might need to execute for each document in your index (you
can try to cache the script filter so it might be faster for subsequent
requests).

Hope this helps as a start for optimization, if not, please provide some
more information.

--Alex

On Sun, Jul 7, 2013 at 2:21 PM, oreno <[hidden email]http://user/SendEmail.jtp?type=node&node=4037659&i=0

wrote:

Hi, I notice that using a script_fields that returns true or false values
is
going much faster then
using the same script but with filter script declaration (so it will
filter
the docs returning false).

I was sure that the filter script is taking so long because I'm using the
source().get(...) method, but turns out that when using the same script,
only with script_fields instead, I'm receiving the performance I need.
the
only problem here is that I want to filter the docs that now have
"MessageReverted" = false.

1.Any way I can filter the docs containing "MessageReverted" = false
?(some
wrapper query?)
2. Any idea way the filter script is taking much longer then the script
field(8000 mill against 250 mill)?

both ways are retrieving the source() for the script logic so it can't be
a
matter of source fetching as far as I understand.

fast:
...,
"script_fields": {
"MessageReverted": {
"script": "revert",
"lang": "native",
"params": {
"startDate": "2013-05-1",
"endDate": "2013-05-1",
"attributeId": "2365443",
"segmentId": "2365443"
}
}
}

slow:
...,
"filter": {
"script": {
"script": "revert",
"lang": "native",
"params": {
"startDate": "2013-05-1",
"endDate": "2013-05-1",
"attributeId": "2365443",
"segmentId": "2365443"
}
}
}

Any idea?

Thanks in advanced,

Oren

--
View this message in context:
http://elasticsearch-users.115913.n3.nabble.com/script-fields-vs-filter-script-tp4037658.html
Sent from the ElasticSearch Users mailing list archive at Nabble.com.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to [hidden email]http://user/SendEmail.jtp?type=node&node=4037659&i=1
.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to [hidden email]http://user/SendEmail.jtp?type=node&node=4037659&i=2
.
For more options, visit https://groups.google.com/groups/opt_out.


If you reply to this email, your message will be added to the discussion
below:

http://elasticsearch-users.115913.n3.nabble.com/script-fields-vs-filter-script-tp4037658p4037659.html
To unsubscribe from script_fields vs filter script, click herehttp://elasticsearch-users.115913.n3.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=4037658&code=b3Jlbm9AZXhlbGF0ZS5jb218NDAzNzY1OHw4ODk1Mjk2Nzg=
.
NAMLhttp://elasticsearch-users.115913.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html!nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers!nabble%3Aemail.naml-instant_emails!nabble%3Aemail.naml-send_instant_email!nabble%3Aemail.naml

--

[image: Inline image 1]
*
*
Oren Orgad | Software Engineer | eXelate http://www.exelate.com/

p: +972- 72-2221329 | e: oreno@exelate.com

signature.bmp (1M) http://elasticsearch-users.115913.n3.nabble.com/attachment/4037661/0/signature.bmp


(kajal) #4

Hey,

Can you actually post your solution If you figured out.
I am having similar issue, I need to filter search result based on
script_field. I don't want to use filter_script though because I am using
facets and I want my records to filter out for facets too.

Do you know if can extends any class or any any plugin or anything to
filter my records based on the script field.

On Sunday, July 7, 2013 1:21:38 PM UTC-4, Oreno wrote:

Hi Alex,
1.I checked the cash solution but its taking 15 times more then my
starting time (10s against 150s), so that will be a problem since my filter
has dynamic params.
It does go fast once it's stored though. Do you know if it's possible to
do some kind of cashing for all source documents for future queries?

2.From what I understand ,both the filter script and the script_field are
suppose to go over each document that results from the prior query.
The only thing I can think of that makes the difference is that the
script_filter actually needs to filter the false documents (for the hit
count) while the script_field only
needs to add the field for the first 10 document returning by default.

I'm trying to figure out how I can speed the response when using source()
on native java script.
I'm assuming the bottle neck is somewhere within creating the response. I
read that using source has some overhead because elasticsearch has to
parse the json source,
but if that was the case here, then I should have received the same big
overhead for both script_field and filter script runs.

All I actually need is the hit count so if I'm correct about the response
parsing and that can be excluded I'll be really glad.

Any idea on the above?

Appreciating your help.

Oren

On Sun, Jul 7, 2013 at 7:13 PM, Alexander Reelsen-2 [via ElasticSearch
Users] <[hidden email]http://user/SendEmail.jtp?type=node&node=4037661&i=0

wrote:

Hey,

what kind of query are you executing? Using script fields results in the
scipt only being executed for each search hit, whereas executing it as a
script filter it might need to execute for each document in your index (you
can try to cache the script filter so it might be faster for subsequent
requests).

Hope this helps as a start for optimization, if not, please provide some
more information.

--Alex

On Sun, Jul 7, 2013 at 2:21 PM, oreno <[hidden email]http://user/SendEmail.jtp?type=node&node=4037659&i=0

wrote:

Hi, I notice that using a script_fields that returns true or false
values is
going much faster then
using the same script but with filter script declaration (so it will
filter
the docs returning false).

I was sure that the filter script is taking so long because I'm using
the
source().get(...) method, but turns out that when using the same script,
only with script_fields instead, I'm receiving the performance I need.
the
only problem here is that I want to filter the docs that now have
"MessageReverted" = false.

1.Any way I can filter the docs containing "MessageReverted" = false
?(some
wrapper query?)
2. Any idea way the filter script is taking much longer then the script
field(8000 mill against 250 mill)?

both ways are retrieving the source() for the script logic so it can't
be a
matter of source fetching as far as I understand.

fast:
...,
"script_fields": {
"MessageReverted": {
"script": "revert",
"lang": "native",
"params": {
"startDate": "2013-05-1",
"endDate": "2013-05-1",
"attributeId": "2365443",
"segmentId": "2365443"
}
}
}

slow:
...,
"filter": {
"script": {
"script": "revert",
"lang": "native",
"params": {
"startDate": "2013-05-1",
"endDate": "2013-05-1",
"attributeId": "2365443",
"segmentId": "2365443"
}
}
}

Any idea?

Thanks in advanced,

Oren

--
View this message in context:
http://elasticsearch-users.115913.n3.nabble.com/script-fields-vs-filter-script-tp4037658.html
Sent from the ElasticSearch Users mailing list archive at Nabble.com.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to [hidden email]http://user/SendEmail.jtp?type=node&node=4037659&i=1
.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to [hidden email]http://user/SendEmail.jtp?type=node&node=4037659&i=2
.
For more options, visit https://groups.google.com/groups/opt_out.


If you reply to this email, your message will be added to the
discussion below:

http://elasticsearch-users.115913.n3.nabble.com/script-fields-vs-filter-script-tp4037658p4037659.html
To unsubscribe from script_fields vs filter script, click here.
NAMLhttp://elasticsearch-users.115913.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html!nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers!nabble%3Aemail.naml-instant_emails!nabble%3Aemail.naml-send_instant_email!nabble%3Aemail.naml

--

[image: Inline image 1]

Oren Orgad | Software Engineer | eXelate http://www.exelate.com/

p: +972- 72-2221329 | e: [hidden email]http://user/SendEmail.jtp?type=node&node=4037661&i=1


View this message in context: Re: script_fields vs filter scripthttp://elasticsearch-users.115913.n3.nabble.com/script-fields-vs-filter-script-tp4037658p4037661.html
Sent from the ElasticSearch Users mailing list archivehttp://elasticsearch-users.115913.n3.nabble.com/at Nabble.com.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/d7f9094f-abb4-42cc-952a-6688166dfaf9%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(brian) #5

I'm trying to filter on a calculated script field as well.

Have you figured this out Kajal?

On Tuesday, May 27, 2014 10:49:35 AM UTC-6, Kajal Patel wrote:

Hey,

Can you actually post your solution If you figured out.
I am having similar issue, I need to filter search result based on
script_field. I don't want to use filter_script though because I am using
facets and I want my records to filter out for facets too.

Do you know if can extends any class or any any plugin or anything to
filter my records based on the script field.

On Sunday, July 7, 2013 1:21:38 PM UTC-4, Oreno wrote:

Hi Alex,
1.I checked the cash solution but its taking 15 times more then my
starting time (10s against 150s), so that will be a problem since my filter
has dynamic params.
It does go fast once it's stored though. Do you know if it's possible to
do some kind of cashing for all source documents for future queries?

2.From what I understand ,both the filter script and the script_field are
suppose to go over each document that results from the prior query.
The only thing I can think of that makes the difference is that the
script_filter actually needs to filter the false documents (for the hit
count) while the script_field only
needs to add the field for the first 10 document returning by default.

I'm trying to figure out how I can speed the response when using source()
on native java script.
I'm assuming the bottle neck is somewhere within creating the response. I
read that using source has some overhead because elasticsearch has to
parse the json source,
but if that was the case here, then I should have received the same big
overhead for both script_field and filter script runs.

All I actually need is the hit count so if I'm correct about the response
parsing and that can be excluded I'll be really glad.

Any idea on the above?

Appreciating your help.

Oren

On Sun, Jul 7, 2013 at 7:13 PM, Alexander Reelsen-2 [via ElasticSearch
Users] <[hidden email]
http://user/SendEmail.jtp?type=node&node=4037661&i=0> wrote:

Hey,

what kind of query are you executing? Using script fields results in the
scipt only being executed for each search hit, whereas executing it as a
script filter it might need to execute for each document in your index (you
can try to cache the script filter so it might be faster for subsequent
requests).

Hope this helps as a start for optimization, if not, please provide some
more information.

--Alex

On Sun, Jul 7, 2013 at 2:21 PM, oreno <[hidden email]
http://user/SendEmail.jtp?type=node&node=4037659&i=0> wrote:

Hi, I notice that using a script_fields that returns true or false
values is
going much faster then
using the same script but with filter script declaration (so it will
filter
the docs returning false).

I was sure that the filter script is taking so long because I'm using
the
source().get(...) method, but turns out that when using the same script,
only with script_fields instead, I'm receiving the performance I need.
the
only problem here is that I want to filter the docs that now have
"MessageReverted" = false.

1.Any way I can filter the docs containing "MessageReverted" = false
?(some
wrapper query?)
2. Any idea way the filter script is taking much longer then the script
field(8000 mill against 250 mill)?

both ways are retrieving the source() for the script logic so it can't
be a
matter of source fetching as far as I understand.

fast:
...,
"script_fields": {
"MessageReverted": {
"script": "revert",
"lang": "native",
"params": {
"startDate": "2013-05-1",
"endDate": "2013-05-1",
"attributeId": "2365443",
"segmentId": "2365443"
}
}
}

slow:
...,
"filter": {
"script": {
"script": "revert",
"lang": "native",
"params": {
"startDate": "2013-05-1",
"endDate": "2013-05-1",
"attributeId": "2365443",
"segmentId": "2365443"
}
}
}

Any idea?

Thanks in advanced,

Oren

--
View this message in context:
http://elasticsearch-users.115913.n3.nabble.com/script-fields-vs-filter-script-tp4037658.html
Sent from the ElasticSearch Users mailing list archive at Nabble.com.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to [hidden email]
http://user/SendEmail.jtp?type=node&node=4037659&i=1.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to [hidden email]
http://user/SendEmail.jtp?type=node&node=4037659&i=2.
For more options, visit https://groups.google.com/groups/opt_out.


If you reply to this email, your message will be added to the
discussion below:

http://elasticsearch-users.115913.n3.nabble.com/script-fields-vs-filter-script-tp4037658p4037659.html
To unsubscribe from script_fields vs filter script, click here.
NAML
http://elasticsearch-users.115913.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html!nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers!nabble%3Aemail.naml-instant_emails!nabble%3Aemail.naml-send_instant_email!nabble%3Aemail.naml

--

[image: Inline image 1]

Oren Orgad | Software Engineer | eXelate http://www.exelate.com/

p: +972- 72-2221329 | e: [hidden email]
http://user/SendEmail.jtp?type=node&node=4037661&i=1


View this message in context: Re: script_fields vs filter script
http://elasticsearch-users.115913.n3.nabble.com/script-fields-vs-filter-script-tp4037658p4037661.html
Sent from the ElasticSearch Users mailing list archive
http://elasticsearch-users.115913.n3.nabble.com/ at Nabble.com.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/1f75923c-6e22-4400-bee2-c4ad11eaeb6d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(system) #6