# How to fetch ~12M documents(may be even more) quickly from ES using scroll API? **URL:** \ **Category:** Elasticsearch\ **Created:** [November 30, 2017, 7:08pm UTC](https://discuss.elastic.co/t/how-to-fetch-12m-documents-may-be-even-more-quickly-from-es-using-scroll-api/109822 "2017-11-30T19:08:07Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![harshil](https://avatars.discourse-cdn.com/v4/letter/h/a4c791/32.png) [@harshil](https://discuss.elastic.co/u/harshil)\ **Post date:** [November 30, 2017, 7:08pm UTC](https://discuss.elastic.co/t/how-to-fetch-12m-documents-may-be-even-more-quickly-from-es-using-scroll-api/109822/1 "2017-11-30T19:08:07Z")
Hello, I have a requirement to fetch ~12M documents (it could get more than that also..). I tried using scroll API with GET {index}/{type}/\_search?scroll=1m { "size": 10000, "sort": ["\_doc"], "\_source": ["field\_1", "field\_2"], "stored\_fields": "\_none", "query": {...} } It takes ~170ms to fetch first page with 10,000 documents with required fields in response. So, to retrieve 12M documents with pagination(10,000 doc) would take ~5min. However, I would like to get this 12M docs as fast as possible. I am thinking to bump up the "index.max\_result\_window" to 100,000 may be ?? Does it even make sense to bump it up the default value? Am I completely off?? What should I do to make ES return these many (~12M) within couple of minutes or even faster. I have 20 data nodes, 3 master nodes, 20 primary side and 1 replica. ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [November 30, 2017, 7:40pm UTC](https://discuss.elastic.co/t/how-to-fetch-12m-documents-may-be-even-more-quickly-from-es-using-scroll-api/109822/2 "2017-11-30T19:40:25Z")
I'm not sure that getting 100000 docs will accelerate things. 170ms to get 10000 documents does not seem that much to me. But may be I'm wrong. Anyway, do you have x-pack monitoring? Anything interesting to see on it? You did not mention BTW how fast is the query by itself? Can you show it? ---
**Author:** ![harshil](https://avatars.discourse-cdn.com/v4/letter/h/a4c791/32.png) [@harshil](https://discuss.elastic.co/u/harshil)\ **Post date:** [November 30, 2017, 8:53pm UTC](https://discuss.elastic.co/t/how-to-fetch-12m-documents-may-be-even-more-quickly-from-es-using-scroll-api/109822/3 "2017-11-30T20:53:01Z")
Hi @dadoonet, Let me correct myself. It takes ~170ms for query I see this `"took": 170,` in a response. I bumped up "index.max\_result\_window" to 20,000 and added more logs and I see page number: 173 -\> Total Hit: 20000 -\> Time took in ms: 74 -\> page time(network+query): 461ms for each scroll request. So total query time is 56sec for all pagination requests. No, I do not have x-pack monitoring. My ES cluster is on AWS, let me check logs/metrics there. But I see the bottleneck here is network (serializer + deserializer + transmit time), which adds another ~4 mins for 12M docs. so, you are right, bumping up "index.max\_result\_window" is not gonna help here. ---
**Author:** ![harshil](https://avatars.discourse-cdn.com/v4/letter/h/a4c791/32.png) [@harshil](https://discuss.elastic.co/u/harshil)\ **Post date:** [November 30, 2017, 11:48pm UTC](https://discuss.elastic.co/t/how-to-fetch-12m-documents-may-be-even-more-quickly-from-es-using-scroll-api/109822/4 "2017-11-30T23:48:49Z")
Hi @dadoonet, Is there a way I can reduce the network latency, by gzipping the response or any other settings? ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [December 28, 2017, 11:49pm UTC](https://discuss.elastic.co/t/how-to-fetch-12m-documents-may-be-even-more-quickly-from-es-using-scroll-api/109822/5 "2017-12-28T23:49:08Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.