# I can't retrieve all data from index

**URL:** <https://discuss.elastic.co/t/i-cant-retrieve-all-data-from-index/97070>\
**Category:** Elasticsearch\
**Created:** [August 15, 2017, 8:03am UTC](https://discuss.elastic.co/t/i-cant-retrieve-all-data-from-index/97070 "2017-08-15T08:03:00Z")\
**Posts on this page:** 16\
**Page:** 1

<div class="post-metadata">

**Author:** ![moabbas](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/moabbas/32/19231_2.png) [@moabbas](https://discuss.elastic.co/u/moabbas)\
**Post date:** [August 15, 2017, 8:03am UTC](https://discuss.elastic.co/t/i-cant-retrieve-all-data-from-index/97070/1 "2017-08-15T08:03:00Z")

</div>

Hello,  
I need to retrieve all the data matching my query, they returned data exceed the default size for window which is 10000,  
How can i do it ?

```
GET logstash-2017.08.04/_search?scroll=5m
{
  "query": { 
  
     "match": {"type": "xms_api_station_info"
        }
   },
   "size": 100000
}

```

Thanks,

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [August 15, 2017, 8:09am UTC](https://discuss.elastic.co/t/i-cant-retrieve-all-data-from-index/97070/2 "2017-08-15T08:09:58Z")

</div>

Have a look at the [scroll API](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-scroll.html). This is specifically designed for retrieving large amounts of data efficiently.

---

<div class="post-metadata">

**Author:** ![moabbas](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/moabbas/32/19231_2.png) [@moabbas](https://discuss.elastic.co/u/moabbas)\
**Post date:** [August 15, 2017, 8:14am UTC](https://discuss.elastic.co/t/i-cant-retrieve-all-data-from-index/97070/3 "2017-08-15T08:14:23Z")

</div>

I used it but it didn't work

> {  
> "error": {  
> "root\_cause": [  
> {  
> "type": "query\_phase\_execution\_exception",  
> "reason": "Batch size is too large, size must be less than or equal to: [10000] but was [1000000]. Scroll batch sizes cost as much memory as result windows so they are controlled by the [index.max\_result\_window] index level setting."  
> }  
> ],  
> "type": "search\_phase\_execution\_exception",  
> "reason": "all shards failed",  
> "phase": "query",  
> "grouped": true,  
> "failed\_shards": [  
> {  
> "shard": 0,  
> "index": "logstash-2017.08.04",  
> "node": "K3CfiisZQpGhffoxkDxR\_A",  
> "reason": {  
> "type": "query\_phase\_execution\_exception",  
> "reason": "Batch size is too large, size must be less than or equal to: [10000] but was [1000000]. Scroll batch sizes cost as much memory as result windows so they are controlled by the [index.max\_result\_window] index level setting."  
> }  
> }  
> ]  
> },  
> "status": 500  
> }

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [August 15, 2017, 8:19am UTC](https://discuss.elastic.co/t/i-cant-retrieve-all-data-from-index/97070/4 "2017-08-15T08:19:06Z")

</div>

You will need to make multiple requests when using the scroll API.

---

<div class="post-metadata">

**Author:** ![moabbas](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/moabbas/32/19231_2.png) [@moabbas](https://discuss.elastic.co/u/moabbas)\
**Post date:** [August 15, 2017, 8:20am UTC](https://discuss.elastic.co/t/i-cant-retrieve-all-data-from-index/97070/5 "2017-08-15T08:20:08Z")

</div>

What you mean by multiple requests ?

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [August 15, 2017, 8:21am UTC](https://discuss.elastic.co/t/i-cant-retrieve-all-data-from-index/97070/6 "2017-08-15T08:21:52Z")

</div>

Did you read the page I linked to?

---

<div class="post-metadata">

**Author:** ![moabbas](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/moabbas/32/19231_2.png) [@moabbas](https://discuss.elastic.co/u/moabbas)\
**Post date:** [August 15, 2017, 8:26am UTC](https://discuss.elastic.co/t/i-cant-retrieve-all-data-from-index/97070/7 "2017-08-15T08:26:32Z")

</div>

Yes, I read it. but i didn't find information about multiple requests

---

<div class="post-metadata">

**Author:** ![moabbas](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/moabbas/32/19231_2.png) [@moabbas](https://discuss.elastic.co/u/moabbas)\
**Post date:** [August 15, 2017, 8:33am UTC](https://discuss.elastic.co/t/i-cant-retrieve-all-data-from-index/97070/8 "2017-08-15T08:33:48Z")

</div>

@Christian_Dahlqvist  
Did you mean multiple requests by doing it in that way ?

> GET /\_search/scroll  
> {  
> "scroll": "5m",  
> "scroll\_id":"DnF1ZXJ5VGhlbkZldGNoBQAAAAAAHvyMFkszQ2ZpaXNaUXBHaGZmb3hrRHhSX0EAAAAAAB78jhZLM0NmaWlzWlFwR2hmZm94a0R4Ul9BAAAAAAAe\_IsWSzNDZmlpc1pRcEdoZmZveGtEeFJfQQAAAAAAHvyNFkszQ2ZpaXNaUXBHaGZmb3hrRHhSX0EAAAAAAB78jxZLM0NmaWlzWlFwR2hmZm94a0R4Ul9B"  
> }

> GET /\_search/scroll  
> {  
> "scroll": "5m",  
> "scroll\_id":"DnF1ZXJ5VGhlbkZldGNoBQAAAAAAHvyMFkszQ2ZpaXNaUXBHaGZmb3hrRHhSX0EAAAAAAB78jhZLM0NmaWlzWlFwR2hmZm94a0R4Ul9BAAAAAAAe\_IsWSzNDZmlpc1pRcEdoZmZveGtEeFJfQQAAAAAAHvyNFkszQ2ZpaXNaUXBHaGZmb3hrRHhSX0EAAAAAAB78jxZLM0NmaWlzWlFwR2hmZm94a0R4Ul9B"  
> }

> GET /\_search/scroll  
> {  
> "scroll": "5m",  
> "scroll\_id":"DnF1ZXJ5VGhlbkZldGNoBQAAAAAAHvyMFkszQ2ZpaXNaUXBHaGZmb3hrRHhSX0EAAAAAAB78jhZLM0NmaWlzWlFwR2hmZm94a0R4Ul9BAAAAAAAe\_IsWSzNDZmlpc1pRcEdoZmZveGtEeFJfQQAAAAAAHvyNFkszQ2ZpaXNaUXBHaGZmb3hrRHhSX0EAAAAAAB78jxZLM0NmaWlzWlFwR2hmZm94a0R4Ul9B"  
> }

> GET /\_search/scroll  
> {  
> "scroll": "5m",  
> "scroll\_id":"DnF1ZXJ5VGhlbkZldGNoBQAAAAAAHvyMFkszQ2ZpaXNaUXBHaGZmb3hrRHhSX0EAAAAAAB78jhZLM0NmaWlzWlFwR2hmZm94a0R4Ul9BAAAAAAAe\_IsWSzNDZmlpc1pRcEdoZmZveGtEeFJfQQAAAAAAHvyNFkszQ2ZpaXNaUXBHaGZmb3hrRHhSX0EAAAAAAB78jxZLM0NmaWlzWlFwR2hmZm94a0R4Ul9B"  
> }

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [August 15, 2017, 9:08am UTC](https://discuss.elastic.co/t/i-cant-retrieve-all-data-from-index/97070/9 "2017-08-15T09:08:43Z")

</div>

Each batch should return a new scroll\_id to be used in the next request. It looks like your example has multiple requests with the same scroll\_id.

---

<div class="post-metadata">

**Author:** ![moabbas](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/moabbas/32/19231_2.png) [@moabbas](https://discuss.elastic.co/u/moabbas)\
**Post date:** [August 15, 2017, 9:16am UTC](https://discuss.elastic.co/t/i-cant-retrieve-all-data-from-index/97070/10 "2017-08-15T09:16:59Z")

</div>

Every request return the same scroll\_id.  
Is that wrong ?

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [August 15, 2017, 9:19am UTC](https://discuss.elastic.co/t/i-cant-retrieve-all-data-from-index/97070/11 "2017-08-15T09:19:57Z")

</div>

Can you show the two first requests you send with their respective responses? If it is too large, store the data externally, e.g. in a gist, and link to it here.

---

<div class="post-metadata">

**Author:** ![moabbas](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/moabbas/32/19231_2.png) [@moabbas](https://discuss.elastic.co/u/moabbas)\
**Post date:** [August 15, 2017, 9:30am UTC](https://discuss.elastic.co/t/i-cant-retrieve-all-data-from-index/97070/12 "2017-08-15T09:30:25Z")

</div>

The returned data aren't the same. I have looked at them if you mean that.

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [August 15, 2017, 9:34am UTC](https://discuss.elastic.co/t/i-cant-retrieve-all-data-from-index/97070/13 "2017-08-15T09:34:11Z")

</div>

So it works for you now?

---

<div class="post-metadata">

**Author:** ![moabbas](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/moabbas/32/19231_2.png) [@moabbas](https://discuss.elastic.co/u/moabbas)\
**Post date:** [August 15, 2017, 9:35am UTC](https://discuss.elastic.co/t/i-cant-retrieve-all-data-from-index/97070/14 "2017-08-15T09:35:04Z")

</div>

Yes, it works for me.  
Thanks a lot for your support, Christian

---

<div class="post-metadata">

**Author:** ![berg](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/berg/32/14962_2.png) [@berg](https://discuss.elastic.co/u/berg)\
**Post date:** [August 21, 2017, 10:57pm UTC](https://discuss.elastic.co/t/i-cant-retrieve-all-data-from-index/97070/15 "2017-08-21T22:57:26Z")

</div>

FWIW, I wrote a tool to extract all data from an index that handles the scroll stuff for you. May not be what you're after but for batch data extraction check out [https://github.com/berglh/escroll](https://github.com/berglh/escroll)

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [September 18, 2017, 10:57pm UTC](https://discuss.elastic.co/t/i-cant-retrieve-all-data-from-index/97070/16 "2017-09-18T22:57:32Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.