Custom score query && pagination


(Yann Barraud) #1

Hi folks,

I was wondering recently : how Elasticsearch perfoms on custom score
queries & pagination ? Does it rescore the whole bunch each time ? Will it
keep custom scores in cache ? Do I have to use a scroll id ?

Thanks.

Regards,
Yann

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(Nick Ethier) #2

Hi Yann,

Regardless of pagination or scrolling, the scores are always computed on
the matching documents, keeping the scores in cache wouldn't be feasible
and would require too much memory.

On the shard level Elasticsearch will build a priority queue. For each
matching document, Elasticsearch computes the score and updates the
priority queue. When finished Elasticsearch takes the first "size" elements
from the queue which are the hits.

Hope this helps!

Cheers.
Nick

On Mon, Nov 18, 2013 at 7:39 AM, Yann Barraud yann.barraud@gmail.comwrote:

Hi folks,

I was wondering recently : how Elasticsearch perfoms on custom score
queries & pagination ? Does it rescore the whole bunch each time ? Will it
keep custom scores in cache ? Do I have to use a scroll id ?

Thanks.

Regards,
Yann

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(Yann Barraud) #3

Hi nick,

I does, for shure !

If I get things well, each page means a whole recalculation ?

Cordialement,
Yann Barraud

2013/11/18 Nick Ethier ncethier@gmail.com

Hi Yann,

Regardless of pagination or scrolling, the scores are always computed on
the matching documents, keeping the scores in cache wouldn't be feasible
and would require too much memory.

On the shard level Elasticsearch will build a priority queue. For each
matching document, Elasticsearch computes the score and updates the
priority queue. When finished Elasticsearch takes the first "size" elements
from the queue which are the hits.

Hope this helps!

Cheers.
Nick

On Mon, Nov 18, 2013 at 7:39 AM, Yann Barraud yann.barraud@gmail.comwrote:

Hi folks,

I was wondering recently : how Elasticsearch perfoms on custom score
queries & pagination ? Does it rescore the whole bunch each time ? Will it
keep custom scores in cache ? Do I have to use a scroll id ?

Thanks.

Regards,
Yann

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/y0eFDXonNWQ/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(Adrien Grand) #4

Hi Yann,

This is correct. The only way to not recompute everything between
successive calls is to use the scan[1] search type (but it is limited: it
doesn't allow for running facets or sorting for example).

[1]
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-request-search-type.html#scan

On Tue, Nov 19, 2013 at 9:35 AM, Yann Barraud yann.barraud@gmail.comwrote:

Hi nick,

I does, for shure !

If I get things well, each page means a whole recalculation ?

Cordialement,
Yann Barraud

2013/11/18 Nick Ethier ncethier@gmail.com

Hi Yann,

Regardless of pagination or scrolling, the scores are always computed on
the matching documents, keeping the scores in cache wouldn't be feasible
and would require too much memory.

On the shard level Elasticsearch will build a priority queue. For each
matching document, Elasticsearch computes the score and updates the
priority queue. When finished Elasticsearch takes the first "size" elements
from the queue which are the hits.

Hope this helps!

Cheers.
Nick

On Mon, Nov 18, 2013 at 7:39 AM, Yann Barraud yann.barraud@gmail.comwrote:

Hi folks,

I was wondering recently : how Elasticsearch perfoms on custom score
queries & pagination ? Does it rescore the whole bunch each time ? Will it
keep custom scores in cache ? Do I have to use a scroll id ?

Thanks.

Regards,
Yann

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/y0eFDXonNWQ/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
Adrien Grand

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(Yann Barraud) #5

Thanks Adrien.

The scroll search won't be better ?

Le mardi 19 novembre 2013 10:58:48 UTC+1, Adrien Grand a écrit :

Hi Yann,

This is correct. The only way to not recompute everything between
successive calls is to use the scan[1] search type (but it is limited: it
doesn't allow for running facets or sorting for example).

[1]
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-request-search-type.html#scan

On Tue, Nov 19, 2013 at 9:35 AM, Yann Barraud <yann.b...@gmail.com<javascript:>

wrote:

Hi nick,

I does, for shure !

If I get things well, each page means a whole recalculation ?

Cordialement,
Yann Barraud

2013/11/18 Nick Ethier <ncet...@gmail.com <javascript:>>

Hi Yann,

Regardless of pagination or scrolling, the scores are always computed on
the matching documents, keeping the scores in cache wouldn't be feasible
and would require too much memory.

On the shard level Elasticsearch will build a priority queue. For each
matching document, Elasticsearch computes the score and updates the
priority queue. When finished Elasticsearch takes the first "size" elements
from the queue which are the hits.

Hope this helps!

Cheers.
Nick

On Mon, Nov 18, 2013 at 7:39 AM, Yann Barraud <yann.b...@gmail.com<javascript:>

wrote:

Hi folks,

I was wondering recently : how Elasticsearch perfoms on custom score
queries & pagination ? Does it rescore the whole bunch each time ? Will it
keep custom scores in cache ? Do I have to use a scroll id ?

Thanks.

Regards,
Yann

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com <javascript:>.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/y0eFDXonNWQ/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearc...@googlegroups.com <javascript:>.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
For more options, visit https://groups.google.com/groups/opt_out.

--
Adrien Grand

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(Adrien Grand) #6

On Tue, Nov 19, 2013 at 11:22 AM, Yann Barraud yann.barraud@gmail.comwrote:

Thanks Adrien.

The scroll search won't be better ?

The purpose of srolling is to execute queries that target different pages
of the same query against the same point-in-time view of the index, so that
results are not duplicated across pages and the number of total hits
remains the same. This doesn't make things less costly. On the contrary,
maintaining the contexts is expensive and we recommend to only use this
feature when having consistent results across pages is a strong
requirement. For example, this shouldn't be used on a high-traffic website
to server user queries.

--
Adrien Grand

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(Yann Barraud) #7

OK, thanks Adrien.

Le mardi 19 novembre 2013 11:33:24 UTC+1, Adrien Grand a écrit :

On Tue, Nov 19, 2013 at 11:22 AM, Yann Barraud <yann.b...@gmail.com<javascript:>

wrote:

Thanks Adrien.

The scroll search won't be better ?

The purpose of srolling is to execute queries that target different pages
of the same query against the same point-in-time view of the index, so that
results are not duplicated across pages and the number of total hits
remains the same. This doesn't make things less costly. On the contrary,
maintaining the contexts is expensive and we recommend to only use this
feature when having consistent results across pages is a strong
requirement. For example, this shouldn't be used on a high-traffic website
to server user queries.

--
Adrien Grand

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(system) #8