Scroll & heap size\ another memory configuration

[liron_gofberg]

Hi,
I want to fetch 50 millions of records into data frame with scan(scroll...)
what heap size do I need for that procedure?
How can I know what is the right size for me?
do I need to configure it only in the master node?
In all the nodes? In the client side?

there is something else that I need to configure to make this procedure running without any problems.?

please..

thanks,
Liron

[warkolm (Mark Walkom)]

What problems are you having?
How big are each scroll sections you are requesting?

[liron_gofberg]

batch size is 1000
and I’ve tried to scroll 11Millions records
smaller queries works fine

I have 3 nodes
heap size at each node was 1.9GB
I changed it to 8GB

I query with routing value so I query from one shard only
Do I need to increase the heap size? what else but heap size can cause memory issue?

Thanks!!

[warkolm (Mark Walkom)]

How big is the total amount of data?

[liron_gofberg]

Now the total data is 20 Millions records(2 shards)
with routing value I query 11M records out of 11M records from one of the two shards

[warkolm (Mark Walkom)]

How big in GB.

[liron_gofberg]

1GB - 5M documents