I have an Elasticsearch (v5.6.10) cluster with 3 nodes.
- Node A : Master
- Node B : Master + Data
- Node C : Master + Data
There are 6 shards per data node with replication set as 1. All 6 primary nodes are in Node B and all 6 replicas are in Node C.
Now suddenly the Node B server got stopped and eventually all the shards in Node C got promoted as primary.
But once Node B came back up, the replica shards are not getting assigned to it. As a result my cluster state is now "Yellow", with 6 unassigned shards.
Also on checking the log in the master node, I found the below errors
[2020-08-04T10:55:05,319][INFO ][o.e.c.r.a.DiskThresholdMonitor] [Bd5E45c] low disk watermark [85%] exceeded on [UBCxWls9R0-WOBhEHMcT5A][UBCxWls][/opt/app/elasticsearch-5.6.10/data/nodes/0] free: 56gb[13.9%], replicas will not be assigned to this node
[2020-08-04T10:55:05,319][INFO ][o.e.c.r.a.DiskThresholdMonitor] [Bd5E45c] low disk watermark [85%] exceeded on [zYg-V-AhSx6J8HyCSsiO5g][zYg-V-A][/opt/app/elasticsearch/data/nodes/0] free: 47.7gb[12.1%], replicas will not be assigned to this node
UBCxWls is Node B
zYg-V-A is Node C
Please help me understand what options I have at this stage.
@DavidTurner You helped me understand https://discuss.elastic.co/t/stop-start-an-elasticsearch-instance-having-all-the-primary-shards/220028. This situation is kind of inline to that. Only the node shutdown was accidental.