I want to have some "cold storage" nodes in my ES cluster, where i will move nodes that are old and rarely searched. (And also never indexed anymore).
I already setup the cluster to have 4 hot servers and 3 cold servers.
The cold servers have tons of storage space, but obviously less RAM.
Specifically, they have 16GB of RAM, with 8GB used for heap.
They store 342 Indices with a total of 1145 shards.
My issue is that some times i'm seeing "OutOfMemoryError: Java heap space" errors in the logs which are causing the cold nodes to go down, and thus making my entire cluster red.
When there is a move of a shard or a recovery, the heap usage is rising to 99% and stays there until eventually there is an OOM error.
How can i handle those situations? I want to keep the cold servers low-end, but still make the indices available.