# Real time scrolling solution **URL:** \ **Category:** Elasticsearch\ **Created:** [July 16, 2018, 3:23pm UTC](https://discuss.elastic.co/t/real-time-scrolling-solution/140177 "2018-07-16T15:23:32Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![TameemSamawi](https://avatars.discourse-cdn.com/v4/letter/t/a3d4f5/32.png) [@TameemSamawi](https://discuss.elastic.co/u/TameemSamawi)\ **Post date:** [July 16, 2018, 3:23pm UTC](https://discuss.elastic.co/t/real-time-scrolling-solution/140177/1 "2018-07-16T15:23:32Z")
The documentation on scrolling explicitly states: > Scrolling is not intended for real time user requests, but rather for processing large amounts of data, e.g. in order to reindex the contents of one index into a new index with a different configuration. What is the recommended solution for real time user requests which return over 10,000 items from ES? ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [July 16, 2018, 7:52pm UTC](https://discuss.elastic.co/t/real-time-scrolling-solution/140177/2 "2018-07-16T19:52:03Z")
What do you mean? I mean what is the use case? Does a user want to see more than 10000 records as fast as possible? Are you looking for something like a Changes API? ---
**Author:** ![TameemSamawi](https://avatars.discourse-cdn.com/v4/letter/t/a3d4f5/32.png) [@TameemSamawi](https://discuss.elastic.co/u/TameemSamawi)\ **Post date:** [July 16, 2018, 8:06pm UTC](https://discuss.elastic.co/t/real-time-scrolling-solution/140177/3 "2018-07-16T20:06:58Z")
The use case: A user sends a custom request to Elasticsearch using the Java API. Over 10,000 records are returned as part of the search response. My current implementation using scrolling takes too long- the query itself runs quickly, but the process of iterating over the scroll response is too slow. I'm sure I can continue to optimize my code, but is there any way around using a scroll response to return over 10,000 documents? And coming back to the scroll response documentation- What is the best practice for returning over 10,000 documents in real-time if scrolling is not intended for real-time requests? The Changes API wouldn't be relevant here, but I haven't heard of that before. I see a Github chain about that but no documentation, is there another link you can send me? ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [July 16, 2018, 8:18pm UTC](https://discuss.elastic.co/t/real-time-scrolling-solution/140177/4 "2018-07-16T20:18:25Z")
Why do you want to get back more than 10000 documents? Sorry I did not get that from your answer. ---
**Author:** ![TameemSamawi](https://avatars.discourse-cdn.com/v4/letter/t/a3d4f5/32.png) [@TameemSamawi](https://discuss.elastic.co/u/TameemSamawi)\ **Post date:** [July 16, 2018, 8:43pm UTC](https://discuss.elastic.co/t/real-time-scrolling-solution/140177/5 "2018-07-16T20:43:32Z")
No worries! A user sends a request for data they need in real-time. i.e. A user needs every single document in our index which matches the termQuery("foo", "bar"). About 3 million documents match that, and every single one of those documents needs to be returned in real-time (or near real-time). Last I remember Elasticsearch caps the number of documents returnable by a searchResponse to 10,000. To go over that limit you need to use scrolling. Scrolling is not intended for real-time user requests (which is clear to me now after seeing how slow it is), but I HAVE to use scrolling if I want to return all 3 million documents. EDIT: Yes I have thought about using several "from/size" queries, but that felt a little hacky. Has the team at ES has dealt with the use case where over 10,000 documents need to be returned in real-time? Our indexing process can be slow, querying is more important. ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [July 16, 2018, 8:55pm UTC](https://discuss.elastic.co/t/real-time-scrolling-solution/140177/6 "2018-07-16T20:55:52Z")
What is unclear to me is what the user will do with 3 millions records on his side. ---
**Author:** ![TameemSamawi](https://avatars.discourse-cdn.com/v4/letter/t/a3d4f5/32.png) [@TameemSamawi](https://discuss.elastic.co/u/TameemSamawi)\ **Post date:** [July 16, 2018, 9:00pm UTC](https://discuss.elastic.co/t/real-time-scrolling-solution/140177/7 "2018-07-16T21:00:50Z")
It's relevant in our use case. I am reading now that we can set the index.max\_result\_window to a value greater than the default of 10,000 which may be the easiest way to handle this use case. Are there any articles or blog posts you can point me to on real-time search optimizations for Elasticsearch? ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [July 16, 2018, 9:19pm UTC](https://discuss.elastic.co/t/real-time-scrolling-solution/140177/8 "2018-07-16T21:19:31Z")
> It's relevant in our use case That's what I want to understand. What is the use case? What does a user do with 3 million records? Increasing the default value will slow down your search, will put a lot of memory pressure on your nodes. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [July 16, 2018, 9:38pm UTC](https://discuss.elastic.co/t/real-time-scrolling-solution/140177/9 "2018-07-16T21:38:02Z")
Using scroll is the best and most efficient way to get large amounts of data out of the cluster. What is the specification of your cluster? What type of storage do you have? What load is the cluster under when you run this? Are you monitoring disk metrics and performance? ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [July 16, 2018, 9:51pm UTC](https://discuss.elastic.co/t/real-time-scrolling-solution/140177/10 "2018-07-16T21:51:25Z")
> [@dadoonet](#): > > That's what I want to understand. What is the use case? What does a user do with 3 million records? > > Increasing the default value will slow down your search, will put a lot of memory pressure on your nodes. To elaborate on this, it's highly unlikely _all 3 million_ results are ever going to be useful. Even google won't return every single possible result available because people rarely look beyond a few pages of results. So if we can understand why this is important then it'll really help with any advice. ---
**Author:** ![TameemSamawi](https://avatars.discourse-cdn.com/v4/letter/t/a3d4f5/32.png) [@TameemSamawi](https://discuss.elastic.co/u/TameemSamawi)\ **Post date:** [July 17, 2018, 2:28pm UTC](https://discuss.elastic.co/t/real-time-scrolling-solution/140177/11 "2018-07-17T14:28:28Z")
Thank you all for your answers. **What does a user do with 3 million documents? What is the use case?** In our use case, the results will be used by a system to perform calculations that are immediately important. The user does nothing with them, a system uses them. Often times, and even in the Engineer training, the use case I've seen for Elasticsearch is a shopping page or an interface in which a user (aka Human) immediately looks at/uses the documents returned. I think that may be why the idea of needing 3 million documents in real-time seems excessive, but I can promise @warkolm that every document is important. Does that help? **What is the specification of your cluster?** The entire cluster is running on default settings. We are not monitoring disk metrics, performance or load. What do you mean by "What storage do you have?" ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [July 17, 2018, 2:32pm UTC](https://discuss.elastic.co/t/real-time-scrolling-solution/140177/12 "2018-07-17T14:32:19Z")
What is the specification of the hosts Elasticsearch is deployed on? Are you using local SSDs, spinning disks or maybe SAN? How many nodes do you have in the cluster? How much data? ---
**Author:** ![TameemSamawi](https://avatars.discourse-cdn.com/v4/letter/t/a3d4f5/32.png) [@TameemSamawi](https://discuss.elastic.co/u/TameemSamawi)\ **Post date:** [July 17, 2018, 3:04pm UTC](https://discuss.elastic.co/t/real-time-scrolling-solution/140177/13 "2018-07-17T15:04:17Z")
Specifications: OS: Windows 10 Enterprise RAM: 32GB Processor: Intel(R) Xeon(R) CPU E3-1535M v5 @2.90GHz System type: 64-bit operating system, x64-based processor Hard Drive: Samsung SSD PM810 2.5" 256GB Data: 25mb - 300mb (varies based on specific solution, but I'll say 25mb if you need one number) Node count: 1 Document count: 85k - 10 million (varies based on specific solution, but I'll say 85k if you need one number) I'm giving you the lower bounds to see if this system can be optimized on those. If so, I'll look at how it performs under a larger load. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [July 17, 2018, 3:19pm UTC](https://discuss.elastic.co/t/real-time-scrolling-solution/140177/14 "2018-07-17T15:19:29Z")
Have you monitored disk performance and CPU usage while running a reasonably large scroll query? ---
**Author:** ![TameemSamawi](https://avatars.discourse-cdn.com/v4/letter/t/a3d4f5/32.png) [@TameemSamawi](https://discuss.elastic.co/u/TameemSamawi)\ **Post date:** [July 17, 2018, 3:28pm UTC](https://discuss.elastic.co/t/real-time-scrolling-solution/140177/15 "2018-07-17T15:28:27Z")
Not yet. Are we talking about using the Nodes stats API to do so? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [July 17, 2018, 3:30pm UTC](https://discuss.elastic.co/t/real-time-scrolling-solution/140177/16 "2018-07-17T15:30:30Z")
No, I do't think that captures disk performance. Not sure how to best get that information on Windows as I am not a Windows user. ---
**Author:** ![TameemSamawi](https://avatars.discourse-cdn.com/v4/letter/t/a3d4f5/32.png) [@TameemSamawi](https://discuss.elastic.co/u/TameemSamawi)\ **Post date:** [July 17, 2018, 3:34pm UTC](https://discuss.elastic.co/t/real-time-scrolling-solution/140177/17 "2018-07-17T15:34:53Z")
Are we trying to determine if this is a resource issue? When I find a way to monitor disk performance and CPU usage on windows, what sort of metrics are you looking for? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [July 17, 2018, 3:39pm UTC](https://discuss.elastic.co/t/real-time-scrolling-solution/140177/18 "2018-07-17T15:39:12Z")
I am wondering if it is a resource issue as you only have one node. It would be interesting to see what the limit is at the moment. Even though you will be able to tune it and scale up/out, retrieving lots of documents from Elasticsearch will probably require a lot of random disk reads and decompression of the source, so I am not sure how fast you will be able to get it. ---
**Author:** ![TameemSamawi](https://avatars.discourse-cdn.com/v4/letter/t/a3d4f5/32.png) [@TameemSamawi](https://discuss.elastic.co/u/TameemSamawi)\ **Post date:** [July 17, 2018, 3:47pm UTC](https://discuss.elastic.co/t/real-time-scrolling-solution/140177/19 "2018-07-17T15:47:08Z")
Gotcha, this use case may be a limitation of Elasticsearch. Let me get those numbers to you within the next week. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [July 17, 2018, 3:48pm UTC](https://discuss.elastic.co/t/real-time-scrolling-solution/140177/20 "2018-07-17T15:48:44Z")
I have not tuned scroll queries, so there may be others with a better idea of what can and can't be achieved and what the limits are. [Next page](https://discuss.elastic.co/t/real-time-scrolling-solution/140177.md?page=2)