Encountered some problems in the horizontal expansion of Elasticsearch

Elastic Stack Elasticsearch
1

My requirement is to do the horizontal expansion of Elasticsearch.
Now I have two nodes and have created some indexes but the capacity is not enough.

I found that after a new node were added into cluster, the shards that originally on the old two nodes were reassigned until all three nodes were distributed with shards.
when the amount of data is large, the time to wait for redistribution is too long.

Here is my question:
1.Is there any way to not change the distribution location of the original old data? After the node is added, only the new index will be allocated on the existing three nodes.

Urge for help! Thanks a lot!

2

Too long for what, may I ask? Or, put differently, why do you need to wait for rebalancing to finish? You can use the cluster normally while this process is ongoing.

Generally you don't want all the new indices on a single new node, because this can cause a load imbalance.

3

But when I adding a new node to a running cluster, the status of cluster is red. Doesn't it means the service is unvaliable?
I mean that it's too long for the cluster's status turns to green, because the shards on the old nodes have to rebalance to all nodes including the new one. And when I searching in the same time, does elasticsearch know the right route to the document considering that the shards are moving?

4

This is unexpected. Adding a new node to a green cluster should leave it green. What is the value of discovery.zen.minimum_master_nodes in the config file of the new node?

5

I didn't configurate that param。
At first, I just have 3 nodes and it seems that the master node is elected automatically. Then I add a node while not configurating the "discovery.zen.minimum_master_nodes" as before.

6

Not setting minimum_master_nodes correctly as you add master-eligible nodes can cause split-brain scenarios. I would recommend reading this documentation on how to set it correctly. If you have 3 master-eligible nodes you need to set it to 2 as having it set to 1 can cause problems. If you add yet another master-eligible node you need to increase it to 3.

7

So If I have 3nodes now and I set minimum_master_nodes=2.
Then I add a new node to this cluster, I set minimum_master_nodes=3 in the configuration file of node4.
Does that be enough? Or I also have to change the configuration file of all old nodes and reload the cluster?

8

I believe you should be able to change it through the cluster update settings API.

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.