High Memory usage even after setting heap memory size

[mayur.buragohain]

I have two 2 clusters with 2 nodes each having 4 cores and 16GB RAM.

Index A( cluster 1) is created with ES 2.4.4 and has about 600GB data across 3 shards.
Index B( cluster 2) is created with ES 5.5.0 and has 6 shards.

The motive is to transfer the data from Index A to index B(newer ES version).

I am using scan-scroll and bulk to transfer data from A to B.

I have set the Heap size on each machine to 8GB using export ES_HEAP_SIZE=8g. I have also verified it using _cat/nodes?h=heap*,name&v.

The transfer works fine initially but after some time, memory usage on the Cluster A machines shows 99% in both nodes of Cluster 1 and scroll throws TransportError "reason":{"type":"search_context_missing_exception","reason":"No search context found for id [275]"}

I am assuming that scroll is not able to fetch results due to slowness of Cluster 1 nodes.

I have checked the Memory usage for the ES java process in node. The commit memory is about 7GB but working set is almost 15GB.

What is the reason for this high memory usage? And how do I overcome this?

[Xavy (Javier)]

Hello:

Some questions still are around my head regarding your issue. Have you checked, when the issue arises, how many documents the index B has?
Have you tried reducing the "size" of the scroll query, and checking if it helps the transfer to finish, or if at least the number of documents when the issue arises increase?

[mayur.buragohain]

Its stop at about 1 hour mark by which about 1.5 million files are usually transferred. I have set the scroll size to 500 and timeout at 30s.
I had started out with size 1000 and default timeout value of 10s to see if it can mitigate the issue but it did not.

[Xavy (Javier)]

Hello mayur:

I actually would suggest checking performance using a lower scroll size (so it does not need to store so much memory for each transfer)

[mayur.buragohain]

Xavy, ES will keep on accumulating the search results in the memory. Reducing the scroll size will only delay the time by when memory is full. It will not prevent it. Correct me if I am wrong.

I tried to disable Paging file but that corrupts the Java settings and i cannot start ES after reboot. I will try to set the Page File to about 2GB and see if that helps.