ElasticSearch cluster restarting/adding node(s)

Elastic Stack Elasticsearch
1

Hello,
i have some questions about how ES should behave while restarting or adding nodes.
CASE 1:
I have 4 nodes in a cluster (every node can be elected as a master and also is a data node).
Number of replicas is set to 0.
Minimum_master_nodes is set to 3.
Let's say I restart one of the nodes (which hold partition_1 data).
While one node is restarting, rest of the nodes will assume that partition_1 is gone.
Question 1.1: Is it possible that rest of nodes will create new partition_1 and after rejoining of restarted node, original/new data from partition_1 will be lost?
Question 1.2: Just to be sure: Let's say restarted node cannot join back to the cluster. So will it reject 'write' operations but will allow 'read' operations?
CASE 2:
I have single-node cluster . I'd like to add a second node. So i change minimum_master_nodes to 2 and start second node with the same configuration (every node can be elected as a master).
Question 2.1: Is it possible that the second (new) node will be elected as a master - if so, what will happen with the data from first (old) node? Will the data be lost/splitted beetwen nodes?
Thank you in advance,
Szymon

2

Hi,

Case 1 - If one of your 4 Nodes is restarted via systemctl/reboot, Elastic will not be able to assume the missing partition_1. You shall see "Unassigned Shard(s)" in this case.

To solve your case, you need to EXLUDE node before restart, in this case, remaining nodes will assume your "partition_1" by reallocating shards correctly between alive nodes

Example for excluding :

curl --user elastic:changeme -XPUT '<my_server_ip>:9200/_cluster/settings' -d '{
  "transient" :{
      "cluster.routing.allocation.exclude._name" : "<my_node_name>"
   }
}'

Case 1 Bis - Yes, you will need (if you do my solution) to include the node, else if your are restarting node, elastic will reassigned missing shards to the right node before the restart .

Case 2 : Yes, if you configured the node 2 to be eligible as Master Node and no, no change will appears the your data.

If all this isn't clear for you, tell me, i would take more times to answer with more details

3

Thank you for answer.
I have some follow-up questions regarding case 1:
Question 1.4:

I don't mind some downtime on part of data (even on whole dataset) but i'd prefer not to reallocate data to other nodes and i'd not like to lose some of data permanently.
What happen after the restarted node joins back to the cluster (Question 1.1) and i didn't use EXLUDE?

Question 1.3: Same situation like in 1.1, but the node is restarted gracefully (by sending SIGTERM to service and start service again). Should I also send EXLUDE? What if there is not enough disc space on remaining nodes to hold data from a node that will restarted?
Thank you in advance,
Szymon

4

Hi,

1.4 - Shards are in Unassigned State, after node restart and back to the cluster.

They are automatically allocated to the right node after this one back online

1.3 - I don't think your node will get out of space, before your restart, elastic allocated space he need for your current (or old) Data. Not sure of my answer but i don't think a node could be in this state when restarting.

That is a good question, i will take times to check it out.

5

Hello,
once again, thanks Ben for quick response.
I'm a little confused. Please let's go again through an example.
"SHARD 0" is in "NODE A".
We kill "NODE A", so "SHARD 0" is now in Unassigned State. No data was transfered/copied/reallocated to another box(es).
"NODE A" is up again, so "SHARD 0" is now assigned to "NODE A". Still, no data was transfered/copied/reallocated to/from another box(es). Am i correct?
So by

did you mean that shards are assigned to right node, but not transered/copied?

And another example:
"SHARD 0" is in "NODE A".
We set "... exclude._name" : "NODE A", so "SHARD 0" is transfered/copied/reallocated to another box(es). But unfortunately, all other nodes all full. What happens with "SHARD 0"? Does it 'stay'
in "NODE A"? If so, what happens after restart - same case as above?

Thank you,
Szymon

6

Hello again :slight_smile:

I just reproduced what your example try to show

Explaining :

4 Nodes :
Node A-01 / Node A-02
Node B-01 / Node A-02

Stopped "brutaly" Node A-01 with systemctl stop elasticsearch-data-01.service (my service name)

Then REPLICAS on otherNODE (like Node-A02) became PRIMARY

Cluster is now YELLOW

Got XX Unassigned shards (From Stopped Node)

Then he "relocate" shards to other nodes (A & B) and he get back to GREEN

When your Node back online he relocate shards again to use your full Data Node in the Cluster.

7

Thank you, but one more thing :slight_smile:
Same situation, but we don't have replicas. How the cluster will behave?

8

Got an idea but i'll test tomorrow :wink:

9

Hi @szymon

I did some test to reply your need.

My Labs :

Node-A01 - Only One Primary Shards & 0 Replicas for Indice

Stopped Node-A01

Capture

Cluster became RED

No relocated shards incoming

10

Hi Ben,
Great, thank you for responses and your time :slight_smile:

11

No problem, here to help Elastic Comminuty :slight_smile:

12

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.