Scrolling is not intended for real time user requests - why?


(Nikita Tovstoles) #1

I am looking at scrollhttp://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-request-scroll.html#search-request-scroll to
support a use case of paging through a list of products resulting from user
checking off a number of product attribute filters in a typical ecommerce
'search & browse' scenario. Think someone wanting to page through a sub-set
of diapers of specific brandshttp://www.amazon.com/s/ref=sr_nr_n_1?rh=n%3A166764011%2Ck%3Adiapers%2Cp_89%3APampers|Diaper+Dude&keywords=diapers&ie=UTF8&qid=1400687984&rnid=2941120011
.

scroll seems like a natural fit there, performing well enough against our
small dataset (master product list on the order of 100K products; user
selections paring down the result set to < 10K) but the docs say not to use
it, seemingly:

"Scrolling is not intended for real time user requests, it is intended for
cases like scrolling over large portions of data that exists within
elasticsearch to reindex it for example."

Does the above warning presume much larger query result sets (and resulting
memory allocation requirements on the ES server)? Or is there another
reason?

thanks,

-nikita

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/8ae9c470-a34a-46da-90fb-409fdfc08e71%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


#2

Nikita,

Pagination is usually implemented using a search request with the 'from' and 'size' parameters [1]. Technically you can do pagination with scroll, as you have seen. However, sooner or later you will want to get more sophisticated in your logic and that will inevitably include sorting and faceting/aggregating, which the scroll API does not support.

Better to start off on a sound footing by using search.

Andrew

  1. http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-request-from-size.html

On May 21, 2014, at 9:09 AM, Nikita Tovstoles nikita.tovstoles@gmail.com wrote:

I am looking at scroll to support a use case of paging through a list of products resulting from user checking off a number of product attribute filters in a typical ecommerce 'search & browse' scenario. Think someone wanting to page through a sub-set of diapers of specific brands.

scroll seems like a natural fit there, performing well enough against our small dataset (master product list on the order of 100K products; user selections paring down the result set to < 10K) but the docs say not to use it, seemingly:

"Scrolling is not intended for real time user requests, it is intended for cases like scrolling over large portions of data that exists within elasticsearch to reindex it for example."

Does the above warning presume much larger query result sets (and resulting memory allocation requirements on the ES server)? Or is there another reason?

thanks,

-nikita

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/8ae9c470-a34a-46da-90fb-409fdfc08e71%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/3859CD3A-BD7C-47FF-A45C-1AFC816E86B4%40elasticsearch.com.
For more options, visit https://groups.google.com/d/optout.


(Nikita Tovstoles) #3

Valid point. However, there does appear to exist a (slight) perf penalty
for retrieving 10,000th item vs. 1st for the same query (ie from=0 vs
from=10000) - about 10x REST latency in my manual tests.

I imagine no such penalty exists with scroll. If so the trade off between
using scroll v. pagination appears to be latency vs flexibility. While
you're probably right wrt pagination being the better approach in 90% of
use cases, in some cases scroll (ex: infinite scroll) appears a better fit.

So, aside from lack of flexibility and higher memory demands, are there
other reasons not to use scroll in real time?

On Wed, May 21, 2014 at 10:17 AM, Andrew Selden <
andrew.selden@elasticsearch.com> wrote:

Nikita,

Pagination is usually implemented using a search request with the ‘from’
and ‘size’ parameters [1]. Technically you can do pagination with scroll,
as you have seen. However, sooner or later you will want to get more
sophisticated in your logic and that will inevitably include sorting and
faceting/aggregating, which the scroll API does not support.

Better to start off on a sound footing by using search.

Andrew

http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-request-from-size.html

On May 21, 2014, at 9:09 AM, Nikita Tovstoles nikita.tovstoles@gmail.com
wrote:

I am looking at scrollhttp://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-request-scroll.html#search-request-scroll to
support a use case of paging through a list of products resulting from user
checking off a number of product attribute filters in a typical ecommerce
'search & browse' scenario. Think someone wanting to page through a sub-set
of diapers of specific brandshttp://www.amazon.com/s/ref=sr_nr_n_1?rh=n%3A166764011%2Ck%3Adiapers%2Cp_89%3APampers|Diaper+Dude&keywords=diapers&ie=UTF8&qid=1400687984&rnid=2941120011
.

scroll seems like a natural fit there, performing well enough against our
small dataset (master product list on the order of 100K products; user
selections paring down the result set to < 10K) but the docs say not to use
it, seemingly:

"Scrolling is not intended for real time user requests, it is intended for
cases like scrolling over large portions of data that exists within
elasticsearch to reindex it for example."

Does the above warning presume much larger query result sets (and
resulting memory allocation requirements on the ES server)? Or is there
another reason?

thanks,

-nikita

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/8ae9c470-a34a-46da-90fb-409fdfc08e71%40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/8ae9c470-a34a-46da-90fb-409fdfc08e71%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/kuyLud4gPrk/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/3859CD3A-BD7C-47FF-A45C-1AFC816E86B4%40elasticsearch.comhttps://groups.google.com/d/msgid/elasticsearch/3859CD3A-BD7C-47FF-A45C-1AFC816E86B4%40elasticsearch.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAJwaA21SSgMgn0ee6bo%3DDck5xzFmwnNvpi7YxS3FbkPjXDGK-g%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


(Jörg Prante) #4

You described a browsing scenario to present products to users. Most users
want to sort the products, for e.g. by price, brands, popularity, or
actuality. With scan/scroll, you can not sort.

Jörg

On Wed, May 21, 2014 at 7:32 PM, Nikita Tovstoles <
nikita.tovstoles@gmail.com> wrote:

Valid point. However, there does appear to exist a (slight) perf penalty
for retrieving 10,000th item vs. 1st for the same query (ie from=0 vs
from=10000) - about 10x REST latency in my manual tests.

I imagine no such penalty exists with scroll. If so the trade off between
using scroll v. pagination appears to be latency vs flexibility. While
you're probably right wrt pagination being the better approach in 90% of
use cases, in some cases scroll (ex: infinite scroll) appears a better fit.

So, aside from lack of flexibility and higher memory demands, are there
other reasons not to use scroll in real time?

On Wed, May 21, 2014 at 10:17 AM, Andrew Selden <
andrew.selden@elasticsearch.com> wrote:

Nikita,

Pagination is usually implemented using a search request with the ‘from’
and ‘size’ parameters [1]. Technically you can do pagination with scroll,
as you have seen. However, sooner or later you will want to get more
sophisticated in your logic and that will inevitably include sorting and
faceting/aggregating, which the scroll API does not support.

Better to start off on a sound footing by using search.

Andrew

http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-request-from-size.html

On May 21, 2014, at 9:09 AM, Nikita Tovstoles nikita.tovstoles@gmail.com
wrote:

I am looking at scrollhttp://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-request-scroll.html#search-request-scroll to
support a use case of paging through a list of products resulting from user
checking off a number of product attribute filters in a typical ecommerce
'search & browse' scenario. Think someone wanting to page through a sub-set
of diapers of specific brandshttp://www.amazon.com/s/ref=sr_nr_n_1?rh=n%3A166764011%2Ck%3Adiapers%2Cp_89%3APampers|Diaper+Dude&keywords=diapers&ie=UTF8&qid=1400687984&rnid=2941120011
.

scroll seems like a natural fit there, performing well enough against our
small dataset (master product list on the order of 100K products; user
selections paring down the result set to < 10K) but the docs say not to use
it, seemingly:

"Scrolling is not intended for real time user requests, it is intended
for cases like scrolling over large portions of data that exists within
elasticsearch to reindex it for example."

Does the above warning presume much larger query result sets (and
resulting memory allocation requirements on the ES server)? Or is there
another reason?

thanks,

-nikita

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/8ae9c470-a34a-46da-90fb-409fdfc08e71%40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/8ae9c470-a34a-46da-90fb-409fdfc08e71%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/kuyLud4gPrk/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.

To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/3859CD3A-BD7C-47FF-A45C-1AFC816E86B4%40elasticsearch.comhttps://groups.google.com/d/msgid/elasticsearch/3859CD3A-BD7C-47FF-A45C-1AFC816E86B4%40elasticsearch.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAJwaA21SSgMgn0ee6bo%3DDck5xzFmwnNvpi7YxS3FbkPjXDGK-g%40mail.gmail.comhttps://groups.google.com/d/msgid/elasticsearch/CAJwaA21SSgMgn0ee6bo%3DDck5xzFmwnNvpi7YxS3FbkPjXDGK-g%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoEXE%3DhifqOW5a4JfLxJrsswGr4k4coJ7pa%2BoGZ9W753sQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


(Nikita Tovstoles) #5

I was able to sort by property while using scroll
On May 21, 2014 11:57 AM, "joergprante@gmail.com" joergprante@gmail.com
wrote:

You described a browsing scenario to present products to users. Most users
want to sort the products, for e.g. by price, brands, popularity, or
actuality. With scan/scroll, you can not sort.

Jörg

On Wed, May 21, 2014 at 7:32 PM, Nikita Tovstoles <
nikita.tovstoles@gmail.com> wrote:

Valid point. However, there does appear to exist a (slight) perf penalty
for retrieving 10,000th item vs. 1st for the same query (ie from=0 vs
from=10000) - about 10x REST latency in my manual tests.

I imagine no such penalty exists with scroll. If so the trade off between
using scroll v. pagination appears to be latency vs flexibility. While
you're probably right wrt pagination being the better approach in 90% of
use cases, in some cases scroll (ex: infinite scroll) appears a better fit.

So, aside from lack of flexibility and higher memory demands, are there
other reasons not to use scroll in real time?

On Wed, May 21, 2014 at 10:17 AM, Andrew Selden <
andrew.selden@elasticsearch.com> wrote:

Nikita,

Pagination is usually implemented using a search request with the ‘from’
and ‘size’ parameters [1]. Technically you can do pagination with scroll,
as you have seen. However, sooner or later you will want to get more
sophisticated in your logic and that will inevitably include sorting and
faceting/aggregating, which the scroll API does not support.

Better to start off on a sound footing by using search.

Andrew

http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-request-from-size.html

On May 21, 2014, at 9:09 AM, Nikita Tovstoles <
nikita.tovstoles@gmail.com> wrote:

I am looking at scrollhttp://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-request-scroll.html#search-request-scroll to
support a use case of paging through a list of products resulting from user
checking off a number of product attribute filters in a typical ecommerce
'search & browse' scenario. Think someone wanting to page through a sub-set
of diapers of specific brandshttp://www.amazon.com/s/ref=sr_nr_n_1?rh=n%3A166764011%2Ck%3Adiapers%2Cp_89%3APampers|Diaper+Dude&keywords=diapers&ie=UTF8&qid=1400687984&rnid=2941120011
.

scroll seems like a natural fit there, performing well enough against
our small dataset (master product list on the order of 100K products; user
selections paring down the result set to < 10K) but the docs say not to use
it, seemingly:

"Scrolling is not intended for real time user requests, it is intended
for cases like scrolling over large portions of data that exists within
elasticsearch to reindex it for example."

Does the above warning presume much larger query result sets (and
resulting memory allocation requirements on the ES server)? Or is there
another reason?

thanks,

-nikita

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/8ae9c470-a34a-46da-90fb-409fdfc08e71%40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/8ae9c470-a34a-46da-90fb-409fdfc08e71%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/kuyLud4gPrk/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.

To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/3859CD3A-BD7C-47FF-A45C-1AFC816E86B4%40elasticsearch.comhttps://groups.google.com/d/msgid/elasticsearch/3859CD3A-BD7C-47FF-A45C-1AFC816E86B4%40elasticsearch.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAJwaA21SSgMgn0ee6bo%3DDck5xzFmwnNvpi7YxS3FbkPjXDGK-g%40mail.gmail.comhttps://groups.google.com/d/msgid/elasticsearch/CAJwaA21SSgMgn0ee6bo%3DDck5xzFmwnNvpi7YxS3FbkPjXDGK-g%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/kuyLud4gPrk/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoEXE%3DhifqOW5a4JfLxJrsswGr4k4coJ7pa%2BoGZ9W753sQ%40mail.gmail.comhttps://groups.google.com/d/msgid/elasticsearch/CAKdsXoEXE%3DhifqOW5a4JfLxJrsswGr4k4coJ7pa%2BoGZ9W753sQ%40mail.gmail.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAJwaA20WKTpcoXRHvNAQJ7F5mremX6pkM-bH7KnnV%3D_BPwxwLA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


(Jörg Prante) #6

I wonder how you did that, the scan search type is defined by doing no sort
and read directly from the shard segments. From the docs:

"The costly part of deep pagination is the global sorting of results, but
if we disable sorting then we can return all documents quite cheaply. To do
this, we use the scan search type. Scan instructs Elasticsearch to do no
sorting, but to just return the next batch of results from every shard
which still has results to return."

http://www.elasticsearch.org/guide/en/elasticsearch/guide/current/scan-scroll.html

Jörg

On Wed, May 21, 2014 at 9:03 PM, Nikita Tovstoles <
nikita.tovstoles@gmail.com> wrote:

I was able to sort by property while using scroll
On May 21, 2014 11:57 AM, "joergprante@gmail.com" joergprante@gmail.com
wrote:

You described a browsing scenario to present products to users. Most
users want to sort the products, for e.g. by price, brands, popularity, or
actuality. With scan/scroll, you can not sort.

Jörg

On Wed, May 21, 2014 at 7:32 PM, Nikita Tovstoles <
nikita.tovstoles@gmail.com> wrote:

Valid point. However, there does appear to exist a (slight) perf penalty
for retrieving 10,000th item vs. 1st for the same query (ie from=0 vs
from=10000) - about 10x REST latency in my manual tests.

I imagine no such penalty exists with scroll. If so the trade off
between using scroll v. pagination appears to be latency vs flexibility.
While you're probably right wrt pagination being the better approach in 90%
of use cases, in some cases scroll (ex: infinite scroll) appears a better
fit.

So, aside from lack of flexibility and higher memory demands, are there
other reasons not to use scroll in real time?

On Wed, May 21, 2014 at 10:17 AM, Andrew Selden <
andrew.selden@elasticsearch.com> wrote:

Nikita,

Pagination is usually implemented using a search request with the
‘from’ and ‘size’ parameters [1]. Technically you can do pagination with
scroll, as you have seen. However, sooner or later you will want to get
more sophisticated in your logic and that will inevitably include sorting
and faceting/aggregating, which the scroll API does not support.

Better to start off on a sound footing by using search.

Andrew

http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-request-from-size.html

On May 21, 2014, at 9:09 AM, Nikita Tovstoles <
nikita.tovstoles@gmail.com> wrote:

I am looking at scrollhttp://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-request-scroll.html#search-request-scroll to
support a use case of paging through a list of products resulting from user
checking off a number of product attribute filters in a typical ecommerce
'search & browse' scenario. Think someone wanting to page through a sub-set
of diapers of specific brandshttp://www.amazon.com/s/ref=sr_nr_n_1?rh=n%3A166764011%2Ck%3Adiapers%2Cp_89%3APampers|Diaper+Dude&keywords=diapers&ie=UTF8&qid=1400687984&rnid=2941120011
.

scroll seems like a natural fit there, performing well enough against
our small dataset (master product list on the order of 100K products; user
selections paring down the result set to < 10K) but the docs say not to use
it, seemingly:

"Scrolling is not intended for real time user requests, it is intended
for cases like scrolling over large portions of data that exists within
elasticsearch to reindex it for example."

Does the above warning presume much larger query result sets (and
resulting memory allocation requirements on the ES server)? Or is there
another reason?

thanks,

-nikita

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/8ae9c470-a34a-46da-90fb-409fdfc08e71%40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/8ae9c470-a34a-46da-90fb-409fdfc08e71%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/kuyLud4gPrk/unsubscribe
.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.

To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/3859CD3A-BD7C-47FF-A45C-1AFC816E86B4%40elasticsearch.comhttps://groups.google.com/d/msgid/elasticsearch/3859CD3A-BD7C-47FF-A45C-1AFC816E86B4%40elasticsearch.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAJwaA21SSgMgn0ee6bo%3DDck5xzFmwnNvpi7YxS3FbkPjXDGK-g%40mail.gmail.comhttps://groups.google.com/d/msgid/elasticsearch/CAJwaA21SSgMgn0ee6bo%3DDck5xzFmwnNvpi7YxS3FbkPjXDGK-g%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/kuyLud4gPrk/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoEXE%3DhifqOW5a4JfLxJrsswGr4k4coJ7pa%2BoGZ9W753sQ%40mail.gmail.comhttps://groups.google.com/d/msgid/elasticsearch/CAKdsXoEXE%3DhifqOW5a4JfLxJrsswGr4k4coJ7pa%2BoGZ9W753sQ%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAJwaA20WKTpcoXRHvNAQJ7F5mremX6pkM-bH7KnnV%3D_BPwxwLA%40mail.gmail.comhttps://groups.google.com/d/msgid/elasticsearch/CAJwaA20WKTpcoXRHvNAQJ7F5mremX6pkM-bH7KnnV%3D_BPwxwLA%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoENUSBmmR%3D7G1M9WpjDqcnUVwaqTnKoYJ%2Bbq8NPOJXtjw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


(Nikita Tovstoles) #7

I used scroll not scan with uri search

http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-request-scroll.html#search-request-scroll
On May 21, 2014 1:04 PM, "joergprante@gmail.com" joergprante@gmail.com
wrote:

I wonder how you did that, the scan search type is defined by doing no
sort and read directly from the shard segments. From the docs:

"The costly part of deep pagination is the global sorting of results, but
if we disable sorting then we can return all documents quite cheaply. To do
this, we use the scan search type. Scan instructs Elasticsearch to do no
sorting, but to just return the next batch of results from every shard
which still has results to return."

http://www.elasticsearch.org/guide/en/elasticsearch/guide/current/scan-scroll.html

Jörg

On Wed, May 21, 2014 at 9:03 PM, Nikita Tovstoles <
nikita.tovstoles@gmail.com> wrote:

I was able to sort by property while using scroll
On May 21, 2014 11:57 AM, "joergprante@gmail.com" joergprante@gmail.com
wrote:

You described a browsing scenario to present products to users. Most
users want to sort the products, for e.g. by price, brands, popularity, or
actuality. With scan/scroll, you can not sort.

Jörg

On Wed, May 21, 2014 at 7:32 PM, Nikita Tovstoles <
nikita.tovstoles@gmail.com> wrote:

Valid point. However, there does appear to exist a (slight) perf
penalty for retrieving 10,000th item vs. 1st for the same query (ie from=0
vs from=10000) - about 10x REST latency in my manual tests.

I imagine no such penalty exists with scroll. If so the trade off
between using scroll v. pagination appears to be latency vs flexibility.
While you're probably right wrt pagination being the better approach in 90%
of use cases, in some cases scroll (ex: infinite scroll) appears a better
fit.

So, aside from lack of flexibility and higher memory demands, are there
other reasons not to use scroll in real time?

On Wed, May 21, 2014 at 10:17 AM, Andrew Selden <
andrew.selden@elasticsearch.com> wrote:

Nikita,

Pagination is usually implemented using a search request with the
‘from’ and ‘size’ parameters [1]. Technically you can do pagination with
scroll, as you have seen. However, sooner or later you will want to get
more sophisticated in your logic and that will inevitably include sorting
and faceting/aggregating, which the scroll API does not support.

Better to start off on a sound footing by using search.

Andrew

http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-request-from-size.html

On May 21, 2014, at 9:09 AM, Nikita Tovstoles <
nikita.tovstoles@gmail.com> wrote:

I am looking at scrollhttp://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-request-scroll.html#search-request-scroll to
support a use case of paging through a list of products resulting from user
checking off a number of product attribute filters in a typical ecommerce
'search & browse' scenario. Think someone wanting to page through a sub-set
of diapers of specific brandshttp://www.amazon.com/s/ref=sr_nr_n_1?rh=n%3A166764011%2Ck%3Adiapers%2Cp_89%3APampers|Diaper+Dude&keywords=diapers&ie=UTF8&qid=1400687984&rnid=2941120011
.

scroll seems like a natural fit there, performing well enough against
our small dataset (master product list on the order of 100K products; user
selections paring down the result set to < 10K) but the docs say not to use
it, seemingly:

"Scrolling is not intended for real time user requests, it is intended
for cases like scrolling over large portions of data that exists within
elasticsearch to reindex it for example."

Does the above warning presume much larger query result sets (and
resulting memory allocation requirements on the ES server)? Or is there
another reason?

thanks,

-nikita

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/8ae9c470-a34a-46da-90fb-409fdfc08e71%40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/8ae9c470-a34a-46da-90fb-409fdfc08e71%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/kuyLud4gPrk/unsubscribe
.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.

To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/3859CD3A-BD7C-47FF-A45C-1AFC816E86B4%40elasticsearch.comhttps://groups.google.com/d/msgid/elasticsearch/3859CD3A-BD7C-47FF-A45C-1AFC816E86B4%40elasticsearch.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAJwaA21SSgMgn0ee6bo%3DDck5xzFmwnNvpi7YxS3FbkPjXDGK-g%40mail.gmail.comhttps://groups.google.com/d/msgid/elasticsearch/CAJwaA21SSgMgn0ee6bo%3DDck5xzFmwnNvpi7YxS3FbkPjXDGK-g%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/kuyLud4gPrk/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoEXE%3DhifqOW5a4JfLxJrsswGr4k4coJ7pa%2BoGZ9W753sQ%40mail.gmail.comhttps://groups.google.com/d/msgid/elasticsearch/CAKdsXoEXE%3DhifqOW5a4JfLxJrsswGr4k4coJ7pa%2BoGZ9W753sQ%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAJwaA20WKTpcoXRHvNAQJ7F5mremX6pkM-bH7KnnV%3D_BPwxwLA%40mail.gmail.comhttps://groups.google.com/d/msgid/elasticsearch/CAJwaA20WKTpcoXRHvNAQJ7F5mremX6pkM-bH7KnnV%3D_BPwxwLA%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/kuyLud4gPrk/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoENUSBmmR%3D7G1M9WpjDqcnUVwaqTnKoYJ%2Bbq8NPOJXtjw%40mail.gmail.comhttps://groups.google.com/d/msgid/elasticsearch/CAKdsXoENUSBmmR%3D7G1M9WpjDqcnUVwaqTnKoYJ%2Bbq8NPOJXtjw%40mail.gmail.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAJwaA23h6fqFwWtqcgXO2n%3DUyf8p-ASg24vjM_MFSwJTKg9QKw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


(Jörg Prante) #8

To initiate scroll search, the scan search type is used. It's in the docs.
Also another disadvantage is in the docs: if you want to provide assistance
to the users browsing for products, for example to break down a large
product set into categories for filtering, you can not offer facets /
aggregates in scan/scroll.

Jörg

On Wed, May 21, 2014 at 10:09 PM, Nikita Tovstoles <
nikita.tovstoles@gmail.com> wrote:

I used scroll not scan with uri search

http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-request-scroll.html#search-request-scroll
On May 21, 2014 1:04 PM, "joergprante@gmail.com" joergprante@gmail.com
wrote:

I wonder how you did that, the scan search type is defined by doing no
sort and read directly from the shard segments. From the docs:

"The costly part of deep pagination is the global sorting of results, but
if we disable sorting then we can return all documents quite cheaply. To do
this, we use the scan search type. Scan instructs Elasticsearch to do no
sorting, but to just return the next batch of results from every shard
which still has results to return."

http://www.elasticsearch.org/guide/en/elasticsearch/guide/current/scan-scroll.html

Jörg

On Wed, May 21, 2014 at 9:03 PM, Nikita Tovstoles <
nikita.tovstoles@gmail.com> wrote:

I was able to sort by property while using scroll
On May 21, 2014 11:57 AM, "joergprante@gmail.com" joergprante@gmail.com
wrote:

You described a browsing scenario to present products to users. Most
users want to sort the products, for e.g. by price, brands, popularity, or
actuality. With scan/scroll, you can not sort.

Jörg

On Wed, May 21, 2014 at 7:32 PM, Nikita Tovstoles <
nikita.tovstoles@gmail.com> wrote:

Valid point. However, there does appear to exist a (slight) perf
penalty for retrieving 10,000th item vs. 1st for the same query (ie from=0
vs from=10000) - about 10x REST latency in my manual tests.

I imagine no such penalty exists with scroll. If so the trade off
between using scroll v. pagination appears to be latency vs flexibility.
While you're probably right wrt pagination being the better approach in 90%
of use cases, in some cases scroll (ex: infinite scroll) appears a better
fit.

So, aside from lack of flexibility and higher memory demands, are
there other reasons not to use scroll in real time?

On Wed, May 21, 2014 at 10:17 AM, Andrew Selden <
andrew.selden@elasticsearch.com> wrote:

Nikita,

Pagination is usually implemented using a search request with the
‘from’ and ‘size’ parameters [1]. Technically you can do pagination with
scroll, as you have seen. However, sooner or later you will want to get
more sophisticated in your logic and that will inevitably include sorting
and faceting/aggregating, which the scroll API does not support.

Better to start off on a sound footing by using search.

Andrew

http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-request-from-size.html

On May 21, 2014, at 9:09 AM, Nikita Tovstoles <
nikita.tovstoles@gmail.com> wrote:

I am looking at scrollhttp://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-request-scroll.html#search-request-scroll to
support a use case of paging through a list of products resulting from user
checking off a number of product attribute filters in a typical ecommerce
'search & browse' scenario. Think someone wanting to page through a sub-set
of diapers of specific brandshttp://www.amazon.com/s/ref=sr_nr_n_1?rh=n%3A166764011%2Ck%3Adiapers%2Cp_89%3APampers|Diaper+Dude&keywords=diapers&ie=UTF8&qid=1400687984&rnid=2941120011
.

scroll seems like a natural fit there, performing well enough against
our small dataset (master product list on the order of 100K products; user
selections paring down the result set to < 10K) but the docs say not to use
it, seemingly:

"Scrolling is not intended for real time user requests, it is
intended for cases like scrolling over large portions of data that exists
within elasticsearch to reindex it for example."

Does the above warning presume much larger query result sets (and
resulting memory allocation requirements on the ES server)? Or is there
another reason?

thanks,

-nikita

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/8ae9c470-a34a-46da-90fb-409fdfc08e71%40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/8ae9c470-a34a-46da-90fb-409fdfc08e71%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in
the Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/kuyLud4gPrk/unsubscribe
.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.

To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/3859CD3A-BD7C-47FF-A45C-1AFC816E86B4%40elasticsearch.comhttps://groups.google.com/d/msgid/elasticsearch/3859CD3A-BD7C-47FF-A45C-1AFC816E86B4%40elasticsearch.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAJwaA21SSgMgn0ee6bo%3DDck5xzFmwnNvpi7YxS3FbkPjXDGK-g%40mail.gmail.comhttps://groups.google.com/d/msgid/elasticsearch/CAJwaA21SSgMgn0ee6bo%3DDck5xzFmwnNvpi7YxS3FbkPjXDGK-g%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/kuyLud4gPrk/unsubscribe
.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoEXE%3DhifqOW5a4JfLxJrsswGr4k4coJ7pa%2BoGZ9W753sQ%40mail.gmail.comhttps://groups.google.com/d/msgid/elasticsearch/CAKdsXoEXE%3DhifqOW5a4JfLxJrsswGr4k4coJ7pa%2BoGZ9W753sQ%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAJwaA20WKTpcoXRHvNAQJ7F5mremX6pkM-bH7KnnV%3D_BPwxwLA%40mail.gmail.comhttps://groups.google.com/d/msgid/elasticsearch/CAJwaA20WKTpcoXRHvNAQJ7F5mremX6pkM-bH7KnnV%3D_BPwxwLA%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/kuyLud4gPrk/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoENUSBmmR%3D7G1M9WpjDqcnUVwaqTnKoYJ%2Bbq8NPOJXtjw%40mail.gmail.comhttps://groups.google.com/d/msgid/elasticsearch/CAKdsXoENUSBmmR%3D7G1M9WpjDqcnUVwaqTnKoYJ%2Bbq8NPOJXtjw%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAJwaA23h6fqFwWtqcgXO2n%3DUyf8p-ASg24vjM_MFSwJTKg9QKw%40mail.gmail.comhttps://groups.google.com/d/msgid/elasticsearch/CAJwaA23h6fqFwWtqcgXO2n%3DUyf8p-ASg24vjM_MFSwJTKg9QKw%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoHfui-66o5wLu6MfXpMNTDh194M8CFdBZrQw04XcQdSww%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


#9

Nikita,

I failed to mention one of the reasons for preferring searching over scrolling. When scrolling, you effectively have something akin to a cursor over a Lucene segment during the life of the scroll. While this resource is in use, ES cannot allow that segment to be merged and reclaimed per the normal segment lifecycle. There is no such open resource in the search case.

Andrew

On May 21, 2014, at 1:09 PM, Nikita Tovstoles nikita.tovstoles@gmail.com wrote:

I used scroll not scan with uri search

http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-request-scroll.html#search-request-scroll

On May 21, 2014 1:04 PM, "joergprante@gmail.com" joergprante@gmail.com wrote:
I wonder how you did that, the scan search type is defined by doing no sort and read directly from the shard segments. From the docs:

"The costly part of deep pagination is the global sorting of results, but if we disable sorting then we can return all documents quite cheaply. To do this, we use the scan search type. Scan instructs Elasticsearch to do no sorting, but to just return the next batch of results from every shard which still has results to return."

http://www.elasticsearch.org/guide/en/elasticsearch/guide/current/scan-scroll.html

Jörg

On Wed, May 21, 2014 at 9:03 PM, Nikita Tovstoles nikita.tovstoles@gmail.com wrote:
I was able to sort by property while using scroll

On May 21, 2014 11:57 AM, "joergprante@gmail.com" joergprante@gmail.com wrote:
You described a browsing scenario to present products to users. Most users want to sort the products, for e.g. by price, brands, popularity, or actuality. With scan/scroll, you can not sort.

Jörg

On Wed, May 21, 2014 at 7:32 PM, Nikita Tovstoles nikita.tovstoles@gmail.com wrote:
Valid point. However, there does appear to exist a (slight) perf penalty for retrieving 10,000th item vs. 1st for the same query (ie from=0 vs from=10000) - about 10x REST latency in my manual tests.

I imagine no such penalty exists with scroll. If so the trade off between using scroll v. pagination appears to be latency vs flexibility. While you're probably right wrt pagination being the better approach in 90% of use cases, in some cases scroll (ex: infinite scroll) appears a better fit.

So, aside from lack of flexibility and higher memory demands, are there other reasons not to use scroll in real time?

On Wed, May 21, 2014 at 10:17 AM, Andrew Selden andrew.selden@elasticsearch.com wrote:
Nikita,

Pagination is usually implemented using a search request with the 'from' and 'size' parameters [1]. Technically you can do pagination with scroll, as you have seen. However, sooner or later you will want to get more sophisticated in your logic and that will inevitably include sorting and faceting/aggregating, which the scroll API does not support.

Better to start off on a sound footing by using search.

Andrew

  1. http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-request-from-size.html

On May 21, 2014, at 9:09 AM, Nikita Tovstoles nikita.tovstoles@gmail.com wrote:

I am looking at scroll to support a use case of paging through a list of products resulting from user checking off a number of product attribute filters in a typical ecommerce 'search & browse' scenario. Think someone wanting to page through a sub-set of diapers of specific brands.

scroll seems like a natural fit there, performing well enough against our small dataset (master product list on the order of 100K products; user selections paring down the result set to < 10K) but the docs say not to use it, seemingly:

"Scrolling is not intended for real time user requests, it is intended for cases like scrolling over large portions of data that exists within elasticsearch to reindex it for example."

Does the above warning presume much larger query result sets (and resulting memory allocation requirements on the ES server)? Or is there another reason?

thanks,

-nikita

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/8ae9c470-a34a-46da-90fb-409fdfc08e71%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/elasticsearch/kuyLud4gPrk/unsubscribe.
To unsubscribe from this group and all its topics, send an email to elasticsearch+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/3859CD3A-BD7C-47FF-A45C-1AFC816E86B4%40elasticsearch.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAJwaA21SSgMgn0ee6bo%3DDck5xzFmwnNvpi7YxS3FbkPjXDGK-g%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/elasticsearch/kuyLud4gPrk/unsubscribe.
To unsubscribe from this group and all its topics, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoEXE%3DhifqOW5a4JfLxJrsswGr4k4coJ7pa%2BoGZ9W753sQ%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAJwaA20WKTpcoXRHvNAQJ7F5mremX6pkM-bH7KnnV%3D_BPwxwLA%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/elasticsearch/kuyLud4gPrk/unsubscribe.
To unsubscribe from this group and all its topics, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoENUSBmmR%3D7G1M9WpjDqcnUVwaqTnKoYJ%2Bbq8NPOJXtjw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAJwaA23h6fqFwWtqcgXO2n%3DUyf8p-ASg24vjM_MFSwJTKg9QKw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/30392DA5-7C12-466C-8A46-474ACC6147B8%40elasticsearch.com.
For more options, visit https://groups.google.com/d/optout.


(Nikita Tovstoles) #10

That's helpful, thanks, Andrew
On May 21, 2014 1:31 PM, "Andrew Selden" andrew.selden@elasticsearch.com
wrote:

Nikita,

I failed to mention one of the reasons for preferring searching over
scrolling. When scrolling, you effectively have something akin to a cursor
over a Lucene segment during the life of the scroll. While this resource is
in use, ES cannot allow that segment to be merged and reclaimed per the
normal segment lifecycle. There is no such open resource in the search case.

Andrew

On May 21, 2014, at 1:09 PM, Nikita Tovstoles nikita.tovstoles@gmail.com
wrote:

I used scroll not scan with uri search

http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-request-scroll.html#search-request-scroll
On May 21, 2014 1:04 PM, "joergprante@gmail.com" joergprante@gmail.com
wrote:

I wonder how you did that, the scan search type is defined by doing no
sort and read directly from the shard segments. From the docs:

"The costly part of deep pagination is the global sorting of results, but
if we disable sorting then we can return all documents quite cheaply. To do
this, we use the scan search type. Scan instructs Elasticsearch to do no
sorting, but to just return the next batch of results from every shard
which still has results to return."

http://www.elasticsearch.org/guide/en/elasticsearch/guide/current/scan-scroll.html

Jörg

On Wed, May 21, 2014 at 9:03 PM, Nikita Tovstoles <
nikita.tovstoles@gmail.com> wrote:

I was able to sort by property while using scroll
On May 21, 2014 11:57 AM, "joergprante@gmail.com" joergprante@gmail.com
wrote:

You described a browsing scenario to present products to users. Most
users want to sort the products, for e.g. by price, brands, popularity, or
actuality. With scan/scroll, you can not sort.

Jörg

On Wed, May 21, 2014 at 7:32 PM, Nikita Tovstoles <
nikita.tovstoles@gmail.com> wrote:

Valid point. However, there does appear to exist a (slight) perf
penalty for retrieving 10,000th item vs. 1st for the same query (ie from=0
vs from=10000) - about 10x REST latency in my manual tests.

I imagine no such penalty exists with scroll. If so the trade off
between using scroll v. pagination appears to be latency vs flexibility.
While you're probably right wrt pagination being the better approach in 90%
of use cases, in some cases scroll (ex: infinite scroll) appears a better
fit.

So, aside from lack of flexibility and higher memory demands, are
there other reasons not to use scroll in real time?

On Wed, May 21, 2014 at 10:17 AM, Andrew Selden <
andrew.selden@elasticsearch.com> wrote:

Nikita,

Pagination is usually implemented using a search request with the
‘from’ and ‘size’ parameters [1]. Technically you can do pagination with
scroll, as you have seen. However, sooner or later you will want to get
more sophisticated in your logic and that will inevitably include sorting
and faceting/aggregating, which the scroll API does not support.

Better to start off on a sound footing by using search.

Andrew

http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-request-from-size.html

On May 21, 2014, at 9:09 AM, Nikita Tovstoles <
nikita.tovstoles@gmail.com> wrote:

I am looking at scrollhttp://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-request-scroll.html#search-request-scroll to
support a use case of paging through a list of products resulting from user
checking off a number of product attribute filters in a typical ecommerce
'search & browse' scenario. Think someone wanting to page through a sub-set
of diapers of specific brandshttp://www.amazon.com/s/ref=sr_nr_n_1?rh=n%3A166764011%2Ck%3Adiapers%2Cp_89%3APampers|Diaper+Dude&keywords=diapers&ie=UTF8&qid=1400687984&rnid=2941120011
.

scroll seems like a natural fit there, performing well enough against
our small dataset (master product list on the order of 100K products; user
selections paring down the result set to < 10K) but the docs say not to use
it, seemingly:

"Scrolling is not intended for real time user requests, it is
intended for cases like scrolling over large portions of data that exists
within elasticsearch to reindex it for example."

Does the above warning presume much larger query result sets (and
resulting memory allocation requirements on the ES server)? Or is there
another reason?

thanks,

-nikita

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/8ae9c470-a34a-46da-90fb-409fdfc08e71%40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/8ae9c470-a34a-46da-90fb-409fdfc08e71%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in
the Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/kuyLud4gPrk/unsubscribe
.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.

To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/3859CD3A-BD7C-47FF-A45C-1AFC816E86B4%40elasticsearch.comhttps://groups.google.com/d/msgid/elasticsearch/3859CD3A-BD7C-47FF-A45C-1AFC816E86B4%40elasticsearch.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAJwaA21SSgMgn0ee6bo%3DDck5xzFmwnNvpi7YxS3FbkPjXDGK-g%40mail.gmail.comhttps://groups.google.com/d/msgid/elasticsearch/CAJwaA21SSgMgn0ee6bo%3DDck5xzFmwnNvpi7YxS3FbkPjXDGK-g%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/kuyLud4gPrk/unsubscribe
.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoEXE%3DhifqOW5a4JfLxJrsswGr4k4coJ7pa%2BoGZ9W753sQ%40mail.gmail.comhttps://groups.google.com/d/msgid/elasticsearch/CAKdsXoEXE%3DhifqOW5a4JfLxJrsswGr4k4coJ7pa%2BoGZ9W753sQ%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAJwaA20WKTpcoXRHvNAQJ7F5mremX6pkM-bH7KnnV%3D_BPwxwLA%40mail.gmail.comhttps://groups.google.com/d/msgid/elasticsearch/CAJwaA20WKTpcoXRHvNAQJ7F5mremX6pkM-bH7KnnV%3D_BPwxwLA%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/kuyLud4gPrk/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoENUSBmmR%3D7G1M9WpjDqcnUVwaqTnKoYJ%2Bbq8NPOJXtjw%40mail.gmail.comhttps://groups.google.com/d/msgid/elasticsearch/CAKdsXoENUSBmmR%3D7G1M9WpjDqcnUVwaqTnKoYJ%2Bbq8NPOJXtjw%40mail.gmail.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAJwaA23h6fqFwWtqcgXO2n%3DUyf8p-ASg24vjM_MFSwJTKg9QKw%40mail.gmail.comhttps://groups.google.com/d/msgid/elasticsearch/CAJwaA23h6fqFwWtqcgXO2n%3DUyf8p-ASg24vjM_MFSwJTKg9QKw%40mail.gmail.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/kuyLud4gPrk/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/30392DA5-7C12-466C-8A46-474ACC6147B8%40elasticsearch.comhttps://groups.google.com/d/msgid/elasticsearch/30392DA5-7C12-466C-8A46-474ACC6147B8%40elasticsearch.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAJwaA23LxOAiM7RhwXWiU5eSaV%2Bmwt6_VScEOS9KO1XDtVoVsA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


(system) #11