Scroll & heap size\ another memory configuration

I want to fetch 50 millions of records into data frame with scan(scroll...)
what heap size do I need for that procedure?
How can I know what is the right size for me?
do I need to configure it only in the master node?
In all the nodes? In the client side?

there is something else that I need to configure to make this procedure running without any problems.?



What problems are you having?
How big are each scroll sections you are requesting?

batch size is 1000
and I’ve tried to scroll 11Millions records
smaller queries works fine

I have 3 nodes
heap size at each node was 1.9GB
I changed it to 8GB

I query with routing value so I query from one shard only
Do I need to increase the heap size? what else but heap size can cause memory issue?


How big is the total amount of data?

Now the total data is 20 Millions records(2 shards)
with routing value I query 11M records out of 11M records from one of the two shards

How big in GB.

1GB - 5M documents

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.