Elasticsearch Cluster Shards rebalancing

Elastic Stack Elasticsearch
1

Hi,

I have few doubts over shards rebalancing.

Case 1:
I have 5 node cluster, 2 data nodes and 3 master nodes. If one data node goes down permanently and I add a different node which contains a different IP.

  1. Will the unassigned shards to assigned to the newly added node?
  2. To achieve the above, what are the configuration changes that I am supposed to do? Do I have to mention the IP of the newly added node in discovery.zen.ping.unicast.hosts?

Case 2 :
Suppose I have 3 node cluster, two data nodes and one master node. Let's say it has around 500 shards in inclusion of primary and replica shards.

  1. When I add a new node,will the shards be rebalanced, by rebalance I mean will the existing shards be assigned to newly added node?
2

Yes.

No, you only need to list the addresses of the master-eligible nodes in discovery.zen.ping.unicast.hosts.

Yes.

(assuming a default configuration, i.e. you have nothing in your configuration that stops the shards being allocated to the new node, such as an allocation filtering rule, and you haven't disabled rebalancing)

3

Do I have to do any other changes in configuration.yml file, with respect to addition of new node in existing cluster?

4

Frequently no, but it depends on exactly what's in the config file. What settings do you think you might need to change?

5

These are normal settings like circuit breakers.
One question out of curiousity:
I have around 1300 shards in my 3 node cluster(2 data and 1 master).
I added fourth node, shards did et rebalance but a very few number of shards i.e., around 20 shards
only got assigned to new node.
Although specifications were similar , why is there a imbalance in assigned number of shards ?

6

1300 shards sounds like a lot for such a small cluster. This article gives some guidance.

Aim to keep the average shard size between at least a few GB and a few tens of GB. For use-cases with time-based data, it is common to see shards between 20GB and 40GB in size.
A good rule-of-thumb is to ensure you keep the number of shards per node below 20 per GB heap it has configured. A node with a 30GB heap should therefore have a maximum of 600 shards, but the further below this limit you can keep it the better.

Recovery speed is limited to ensure that the cluster remains stable, and only occurs if the cluster health is green. My guess is that the rebalancing was still ongoing, or else the cluster stopped being green.

7

Your guess was right David. Shards got balanced after some period of time
Thanks for your help

1 Like
8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.