Data nodes HA and Shards allocation/creation

Elastic Stack Elasticsearch
1

If we have two data nodes and replica has been set to 1, and if a node down whether the shards will be created in other node newly and will require the same amount of additional disk space which has been lost due to the unavailability of the other node?

Also what happens when the other node comes up and the shards have been synchronized to have the latest data, whether the newly created shards in other node will be deleted and will free up the disk space?

2

Elasticsearch will not allocate two copies of a shard to the same node. To do so would cost resources (particularly disk space and IO) without giving any resilience benefit. In a two-node setup if one of the nodes becomes unavailable then all the replica shards will simply be unassigned and your cluster health will show as YELLOW. When the other node returns the replicas will be allocated to it again and will sync up with their primaries.

3

Thanks. Would like to know whether it will be the same behavior if 3 or more nodes are there and one node goes down.

4

Roughly speaking, no. The master will try and allocate any unallocated shards to bring your cluster back into a resilient state; in the 2-node case the rule "do not allocate more than one copy of a shard to a node" prevents it from doing so, but if there is somewhere to allocate the unallocated shards then it will use it.

However there are features such as delayed allocation to prevent this from happening immediately, in case the node comes back quickly, and various other settings that control how the shard recoveries take place, how many recoveries can happen concurrently and so on.

5

Thank you. How the behavior will be in case of hot and warm data nodes architecture where 2 hot nodes and 2 warm nodes are present? Weather they will work in isolation in case of one node goes in each type?

6

The hot/warm architecture uses allocation filtering, which imposes other strict rules that the master will obey when it is trying to find somewhere to allocate shards alongside the "do not allocate more than one copy of a shard to a node" rule. So the shards of a hot index will only be allocated to hot nodes and similarly for the warm ones, and if you lose a warm node then its shards will remain unallocated until the node comes back online and will not be allocated to one of the hot nodes.

1 Like
7

Thanks for the clarification. In a 3 data node scenario one data node goes down and after allocating the shards from the node which went down to the available data nodes the cluster status will turn back to GREEN from YELLOW. After this when the node which was down is back up whether the shards which have been allocated on other nodes will be deleted hence freeing up the disk space?

8

Pretty much, yes. I'm not sure if that is guaranteed to happen immediately, because the cluster may decide to rebalance by relocating some shards onto the newly-restored node first, and it can speed up these recoveries if it can make use of any data that's already there. Eventually it'll settle down and delete any unnecessary shard copies to free up their disk space.

If no such rebalancing takes place (e.g. it's disabled) and the cluster health is GREEN then any unnecessary shard copies will be removed, freeing up the disk space, pretty soon.

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.