# Memory error when using scan In python to retrieve millions of documents

**URL:** <https://discuss.elastic.co/t/memory-error-when-using-scan-in-python-to-retrieve-millions-of-documents/241615>\
**Category:** Elasticsearch\
**Created:** [July 17, 2020, 8:54am UTC](https://discuss.elastic.co/t/memory-error-when-using-scan-in-python-to-retrieve-millions-of-documents/241615 "2020-07-17T08:54:13Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![liron\_gofberg](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/liron_gofberg/32/59823_2.png) [@liron\_gofberg](https://discuss.elastic.co/u/liron_gofberg)\
**Post date:** [July 17, 2020, 8:54am UTC](https://discuss.elastic.co/t/memory-error-when-using-scan-in-python-to-retrieve-millions-of-documents/241615/1 "2020-07-17T08:54:13Z")

</div>

I’m using scan in python to retrieve my data  
It is working when I ask for thousands of documents but when I ask for millions of records I get an error after a while and that error relate to the memory

What could be happen?

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [July 17, 2020, 9:03am UTC](https://discuss.elastic.co/t/memory-error-when-using-scan-in-python-to-retrieve-millions-of-documents/241615/2 "2020-07-17T09:03:52Z")

</div>

Please share the full error.  
It'd also help if you shared the code that you are using.

---

<div class="post-metadata">

**Author:** ![liron\_gofberg](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/liron_gofberg/32/59823_2.png) [@liron\_gofberg](https://discuss.elastic.co/u/liron_gofberg)\
**Post date:** [July 17, 2020, 6:42pm UTC](https://discuss.elastic.co/t/memory-error-when-using-scan-in-python-to-retrieve-millions-of-documents/241615/3 "2020-07-17T18:42:32Z")

</div>

Thanks for answering

scan use scroll  
scroll raise the error

I don’t have the code now but it doesn’t matter  
I would like you to think about what could happen If we talk about memory error  
the error happens when I'm trying to fetch multiple millions of records  
but if I ask for 500,000 everything is ok, there is no error

I read about the need to set heap size

If I query with scroll millions of records from Elasticsearch  
do I need to set heap size of the master node only? do I need to configure the data nodes too?  
do I need to do something else?

the master node need to have available space to assemble the results of all the shards  
so if I ask for 20GB of results, The master node need to have 20 GB available?(In heap size?)

I would like to know what memory scroll need to function right with millions of records

Please

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [August 14, 2020, 6:42pm UTC](https://discuss.elastic.co/t/memory-error-when-using-scan-in-python-to-retrieve-millions-of-documents/241615/4 "2020-08-14T18:42:56Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.