# How to retrieve all records from index

**URL:** <https://discuss.elastic.co/t/how-to-retrieve-all-records-from-index/156668>\
**Category:** Elasticsearch\
**Created:** [November 14, 2018, 1:22pm UTC](https://discuss.elastic.co/t/how-to-retrieve-all-records-from-index/156668 "2018-11-14T13:22:55Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![Mahmoodm](https://avatars.discourse-cdn.com/v4/letter/m/858c86/32.png) [@Mahmoodm](https://discuss.elastic.co/u/Mahmoodm)\
**Post date:** [November 14, 2018, 1:22pm UTC](https://discuss.elastic.co/t/how-to-retrieve-all-records-from-index/156668/1 "2018-11-14T13:22:55Z")

</div>

Hi  
I am trying to retrieve all records from elastic search index  
import elasticsearch  
from elasticsearch import Elasticsearch  
ret\_size = 0  
cluster = "dvfarmil"  
es = Elasticsearch([{'host': 'YYYYYYY', 'port': 80}])  
res = es.search(index='XXXXXXXXXX',  
scroll = '50m',  
body={"size":10000, "query": {"bool": {"must": [{"match": {"clustername":"OOOOO"}}]}}})  
sid=res['\_scroll\_id']  
while ( ret\_size \> 0 ):  
res = es.scroll (scroll\_id=sid,scroll = '50m')  
ret\_size += len(res['hits']['hits'])  
if ( sid == res['\_scroll\_id'] ):  
exit(0)

```
else:
    sid=res['_scroll_id']

```

scroll\_id isn't being changed and I know that the index contain ten thousands of records  
please advise  
Mahmood

---

<div class="post-metadata">

**Author:** ![fardhana](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/fardhana/32/37533_2.png) [@fardhana](https://discuss.elastic.co/u/fardhana)\
**Post date:** [November 14, 2018, 1:43pm UTC](https://discuss.elastic.co/t/how-to-retrieve-all-records-from-index/156668/2 "2018-11-14T13:43:26Z")

</div>

Do you mean listing all document without pagination?  
you can recognise number of document in your cluster based on `hits.total` right?

---

<div class="post-metadata">

**Author:** ![Mahmoodm](https://avatars.discourse-cdn.com/v4/letter/m/858c86/32.png) [@Mahmoodm](https://discuss.elastic.co/u/Mahmoodm)\
**Post date:** [November 15, 2018, 8:35am UTC](https://discuss.elastic.co/t/how-to-retrieve-all-records-from-index/156668/3 "2018-11-15T08:35:47Z")

</div>

> [@fardhana](#):
>
> hits.total

I want to retrieve all docs, I don't how much are they but for sure they are more than 300,000 docs  
what the correct way to do it?  
what I have done isn't working

---

<div class="post-metadata">

**Author:** ![fardhana](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/fardhana/32/37533_2.png) [@fardhana](https://discuss.elastic.co/u/fardhana)\
**Post date:** [November 15, 2018, 9:04am UTC](https://discuss.elastic.co/t/how-to-retrieve-all-records-from-index/156668/4 "2018-11-15T09:04:11Z")

</div>

I prefer to run this query `{ "query": { "match_all": {} }, "size": 5000, "from": 0 }`  
then i will doing loop for next page (by modifiying `from` clause) for next page.

Anyway, you can check how much the exact total docs you have in the response `{ "took": 94, "timed_out": false, "_shards": { "total": 1, "successful": 1, "skipped": 0, "failed": 0 }, "hits": { "total": 852705, "max_score": 1,...}}` in my case i have **852705** document

---

<div class="post-metadata">

**Author:** ![Mahmoodm](https://avatars.discourse-cdn.com/v4/letter/m/858c86/32.png) [@Mahmoodm](https://discuss.elastic.co/u/Mahmoodm)\
**Post date:** [November 15, 2018, 12:09pm UTC](https://discuss.elastic.co/t/how-to-retrieve-all-records-from-index/156668/5 "2018-11-15T12:09:10Z")

</div>

something isn't working, total is 2033951  
body={"size":10000,"from":0, "query": {"bool": {"must": [{"match": {"clustername":"test1"}}]}}}) // worked  
body={"size":10000,"from":10001, "query": {"bool": {"must": [{"match": {"clustername":"test1"}}]}}}) // failed

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [December 13, 2018, 12:09pm UTC](https://discuss.elastic.co/t/how-to-retrieve-all-records-from-index/156668/6 "2018-12-13T12:09:11Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.