# Fetching results beyond 10k - scroll/searchAfter **URL:** \ **Category:** Elasticsearch\ **Created:** [July 27, 2021, 8:02am UTC](https://discuss.elastic.co/t/fetching-results-beyond-10k-scroll-searchafter/279702 "2021-07-27T08:02:17Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![mouli\_99](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mouli_99/32/79243_2.png) [@mouli\_99](https://discuss.elastic.co/u/mouli_99)\ **Post date:** [July 27, 2021, 8:02am UTC](https://discuss.elastic.co/t/fetching-results-beyond-10k-scroll-searchafter/279702/1 "2021-07-27T08:02:17Z")
We are on elastic 7.8, we have use case to fetch beyond 10k results are in interactive mode from users on pretty decent scale (say ~40 concurrent hits) We have stumbled upon 1. scroll (everytime open a scroll, retrieve and close the scroll) 2. searchAfter (sorted by \_id ) We did Load Testing with above both options Config: 1. Data Nodes -6 (4cores, 64G RAM, 1TB) 2. Co-ordinator - 2 (4cores, 32G RAM) 3. 60 Concurrent Threads doing search on cluster retrieving top 5000 results 4. Index - 20M docs, 30 shards 2 replicas Observations: 1. Search is really bad with latencies over 40 secs 2. With scrolls/searchAfter even regular searches are getting slower 3. If disable scrolls/searchAfter, regular searches are getting better ~3-4 secs under load 4. We dont see CPU/RAM being spiked up Questions: 1. Can someone throw light how scrolls/searchAfter are so expensive 2. As our resources are under utilized, can someone point us to settings to be changes from default which are otherwise defaulted to conservative numbers? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [July 27, 2021, 8:34am UTC](https://discuss.elastic.co/t/fetching-results-beyond-10k-scroll-searchafter/279702/2 "2021-07-27T08:34:06Z")
> [@mouli\_99](#): > > As our resources are under utilized Elasticsearch is not always limited by CPU, and can often instead be limited by disk I/O. What kind of storage do you have? Have you monitored disk I/o and iowait? ---
**Author:** ![mouli\_99](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mouli_99/32/79243_2.png) [@mouli\_99](https://discuss.elastic.co/u/mouli_99)\ **Post date:** [July 27, 2021, 9:08am UTC](https://discuss.elastic.co/t/fetching-results-beyond-10k-scroll-searchafter/279702/3 "2021-07-27T09:08:02Z")
dont see much spikes in IO read/Writes as well, we are on ssd 1TB local storage per node ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [July 27, 2021, 9:30am UTC](https://discuss.elastic.co/t/fetching-results-beyond-10k-scroll-searchafter/279702/4 "2021-07-27T09:30:33Z")
How large is the index (in GB)? What is the average shard size? ---
**Author:** ![mouli\_99](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mouli_99/32/79243_2.png) [@mouli\_99](https://discuss.elastic.co/u/mouli_99)\ **Post date:** [July 27, 2021, 10:02am UTC](https://discuss.elastic.co/t/fetching-results-beyond-10k-scroll-searchafter/279702/5 "2021-07-27T10:02:23Z")
~500G for entire index and ~8G per shard ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [August 24, 2021, 10:02am UTC](https://discuss.elastic.co/t/fetching-results-beyond-10k-scroll-searchafter/279702/6 "2021-08-24T10:02:36Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.