# Scrolling performance
**URL:** https://discuss.elastic.co/t/scrolling-performance/5958
**Category:** Elasticsearch
**Created:** [November 23, 2011, 9:47pm UTC](https://discuss.elastic.co/t/scrolling-performance/5958 "2011-11-23T21:47:08Z")
**Posts on this page:** 6
**Page:** 1
### Author:  [@Josh\_Holtzman](https://discuss.elastic.co/u/Josh_Holtzman)
#### Post date: [November 23, 2011, 9:47pm UTC](https://discuss.elastic.co/t/scrolling-performance/5958/1 "2011-11-23T21:47:08Z")
I'm trying to implement an index exporter by doing a "match\_all" query
and scrolling through the entire index, 100 or 1000 documents at a
time. I'm seeing a significant slowdown in scrolling over time. The
first scroll via the rest api returns in \< 50ms, but once I've
scrolled through 1.5 million of the 2 million total docs, the time to
execute is \> 1 second. I've set the scroll timeout to 10 seconds,
which performs better than 10 minutes, but I can't decrease the
timeout much more without risking timing out between calls.
I'm wondering if a) this dramatic slowdown is expected, and b) if
there's a better way to scroll through all documents quickly.
Thanks,
Josh
---
### Author:  [@Josh\_Holtzman](https://discuss.elastic.co/u/Josh_Holtzman)
#### Post date: [November 23, 2011, 9:49pm UTC](https://discuss.elastic.co/t/scrolling-performance/5958/2 "2011-11-23T21:49:55Z")
Sorry, I missed this thread, which covers the same topic:
[http://groups.google.com/group/elasticsearch/browse\_thread/thread/667811af4a1f5fae](http://groups.google.com/group/elasticsearch/browse_thread/thread/667811af4a1f5fae)
On Nov 23, 1:47 pm, jholtzman [jmholtz...@gmail.com](mailto:jmholtz...@gmail.com) wrote:
> I'm trying to implement an index exporter by doing a "match\_all" query
> and scrolling through the entire index, 100 or 1000 documents at a
> time. I'm seeing a significant slowdown in scrolling over time. The
> first scroll via the rest api returns in \< 50ms, but once I've
> scrolled through 1.5 million of the 2 million total docs, the time to
> execute is \> 1 second. I've set the scroll timeout to 10 seconds,
> which performs better than 10 minutes, but I can't decrease the
> timeout much more without risking timing out between calls.
>
> I'm wondering if a) this dramatic slowdown is expected, and b) if
> there's a better way to scroll through all documents quickly.
>
> Thanks,
> Josh
---
### Author:  [@kimchy](https://discuss.elastic.co/u/kimchy)
#### Post date: [November 24, 2011, 2:13pm UTC](https://discuss.elastic.co/t/scrolling-performance/5958/3 "2011-11-24T14:13:44Z")
- Setting the a lower scroll timeout will not affect the performance of
scrolling.
- Strange that it ends up being 1 second for the 1.5M scroll on match\_all,
how many shards do you have in the index? How many nodes in the cluster?
On Wed, Nov 23, 2011 at 11:49 PM, jholtzman [jmholtzman@gmail.com](mailto:jmholtzman@gmail.com) wrote:
> Sorry, I missed this thread, which covers the same topic:
>
> [http://groups.google.com/group/elasticsearch/browse\_thread/thread/667811af4a1f5fae](http://groups.google.com/group/elasticsearch/browse_thread/thread/667811af4a1f5fae)
>
> On Nov 23, 1:47 pm, jholtzman [jmholtz...@gmail.com](mailto:jmholtz...@gmail.com) wrote:
>
> > I'm trying to implement an index exporter by doing a "match\_all" query
> > and scrolling through the entire index, 100 or 1000 documents at a
> > time. I'm seeing a significant slowdown in scrolling over time. The
> > first scroll via the rest api returns in \< 50ms, but once I've
> > scrolled through 1.5 million of the 2 million total docs, the time to
> > execute is \> 1 second. I've set the scroll timeout to 10 seconds,
> > which performs better than 10 minutes, but I can't decrease the
> > timeout much more without risking timing out between calls.
> >
> > I'm wondering if a) this dramatic slowdown is expected, and b) if
> > there's a better way to scroll through all documents quickly.
> >
> > Thanks,
> > Josh
---
### Author:  [@Josh\_Holtzman](https://discuss.elastic.co/u/Josh_Holtzman)
#### Post date: [November 30, 2011, 10:50pm UTC](https://discuss.elastic.co/t/scrolling-performance/5958/4 "2011-11-30T22:50:49Z")
4 shards, all on a single node.
Thanks,
Josh
On Nov 24, 6:13 am, Shay Banon [kim...@gmail.com](mailto:kim...@gmail.com) wrote:
> - Setting the a lower scroll timeout will not affect the performance of
> scrolling.
> - Strange that it ends up being 1 second for the 1.5M scroll on match\_all,
> how many shards do you have in the index? How many nodes in the cluster?
>
> On Wed, Nov 23, 2011 at 11:49 PM, jholtzman [jmholtz...@gmail.com](mailto:jmholtz...@gmail.com) wrote:
>
> > Sorry, I missed this thread, which covers the same topic:
>
> > [http://groups.google.com/group/elasticsearch/browse\_thread/thread/667](http://groups.google.com/group/elasticsearch/browse_thread/thread/667)...
>
> > On Nov 23, 1:47 pm, jholtzman [jmholtz...@gmail.com](mailto:jmholtz...@gmail.com) wrote:
> >
> > > I'm trying to implement an index exporter by doing a "match\_all" query
> > > and scrolling through the entire index, 100 or 1000 documents at a
> > > time. I'm seeing a significant slowdown in scrolling over time. The
> > > first scroll via the rest api returns in \< 50ms, but once I've
> > > scrolled through 1.5 million of the 2 million total docs, the time to
> > > execute is \> 1 second. I've set the scroll timeout to 10 seconds,
> > > which performs better than 10 minutes, but I can't decrease the
> > > timeout much more without risking timing out between calls.
>
> > > I'm wondering if a) this dramatic slowdown is expected, and b) if
> > > there's a better way to scroll through all documents quickly.
>
> > > Thanks,
> > > Josh
---
### Author:  [@Josh\_Holtzman](https://discuss.elastic.co/u/Josh_Holtzman)
#### Post date: [December 16, 2011, 12:24am UTC](https://discuss.elastic.co/t/scrolling-performance/5958/5 "2011-12-16T00:24:44Z")
The documentation at [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/api/search/scroll.html)
describes scrolling through a large set of data using this URL as an
example: [http://localhost:9200/twitter/tweet/\_search?scroll=5m](http://localhost:9200/twitter/tweet/_search?scroll=5m)
This unfortunately excludes the key parameter: search\_type=scan.
Following the documentation at [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/api/search/search-type.html)
did the trick, and now the scrolling performance is once again
constant across requests.
Thanks,
Josh
On Nov 30, 2:50 pm, jholtzman [jmholtz...@gmail.com](mailto:jmholtz...@gmail.com) wrote:
> 4 shards, all on a single node.
>
> Thanks,
> Josh
>
> On Nov 24, 6:13 am, Shay Banon [kim...@gmail.com](mailto:kim...@gmail.com) wrote:
>
> > - Setting the a lower scroll timeout will not affect the performance of
> > scrolling.
> > - Strange that it ends up being 1 second for the 1.5M scroll on match\_all,
> > how many shards do you have in the index? How many nodes in the cluster?
>
> > On Wed, Nov 23, 2011 at 11:49 PM,jholtzman[jmholtz...@gmail.com](mailto:jmholtz...@gmail.com) wrote:
> >
> > > Sorry, I missed this thread, which covers the same topic:
>
> > > [http://groups.google.com/group/elasticsearch/browse\_thread/thread/667](http://groups.google.com/group/elasticsearch/browse_thread/thread/667)...
>
> > > On Nov 23, 1:47 pm,jholtzman[jmholtz...@gmail.com](mailto:jmholtz...@gmail.com) wrote:
> > >
> > > > I'm trying to implement an index exporter by doing a "match\_all" query
> > > > and scrolling through the entire index, 100 or 1000 documents at a
> > > > time. I'm seeing a significant slowdown in scrolling over time. The
> > > > first scroll via the rest api returns in \< 50ms, but once I've
> > > > scrolled through 1.5 million of the 2 million total docs, the time to
> > > > execute is \> 1 second. I've set the scroll timeout to 10 seconds,
> > > > which performs better than 10 minutes, but I can't decrease the
> > > > timeout much more without risking timing out between calls.
>
> > > > I'm wondering if a) this dramatic slowdown is expected, and b) if
> > > > there's a better way to scroll through all documents quickly.
>
> > > > Thanks,
> > > > Josh
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 6, 2017, 3:45am UTC](https://discuss.elastic.co/t/scrolling-performance/5958/6 "2017-07-06T03:45:18Z")