Currently we have one-node-cluster. This node has high heap usage (from time to time we even see "circuit breaking exception") and high disk usage (currently over high disk watermark value) because it keep lots of indices.
We plan to offload this node by adding another one.
From our hardware limitation new node could have maximum:
- 50% of RAM of the current node (so also 50% of heap size of the current node)
- 40% of storage capacity of the current node
We worries after adding another node the cluster rebalancing algorithm will starts migrate indices (shards) and completely fill new node's heap before achieve stop balancing threshold (probably it's depends of shards count on node or node's disk usage).
And finally our cluster stops working because of new node's heap full, so will be worst as now
Could you tell me what parameters cluster's rebalancing algorithm take account?
Is this algorithm checks heap size on node during rebalancing or only concentrate about disk usage and shards number parameters?
Are our worries justified?