Rebuild Cluster - Unassigned shards

Elastic Stack Elasticsearch
1

Hi Team,
It's urgent, though i doubt about recovery but if you can help it will be great.

My ES cluster datanodes are under autoscaling group and while updating the stack we did a mistake. The result is, we had 8 data nodes out of which 4 got terminated. New nodes came up but now ES is RED because shards (450) are unassigned.

To start ELK i closed all indexes for which shards were unassigned but need to reproduce the data. Snapshot i tested but not yet implemented, so backup looks like not an option.

Here is some conf details:
ES Cluster: 1-master, 1-client,8-data node
Conf: 4-shards and 2-replicas
Size: 150GB (before incident)

Please let me know if you need more information.

Thanks & Regards,

2

If you have 3 copies of each shard (1 primary and 2 replicas) and have not used shard allocation awareness to ensure that at least one replica for each shard is present on the data nodes that remain, it is possible that some shards may have had all copies allocated to the nodes that were terminated. If this is the case and you do not have a snapshot to restore from, you will either need to bring the nodes you terminated back up again (if that is even possible) or reindex your data from an external source.

3

Do you different ES versions across your cluster ?

4

No, versions are same.

I will try rebuilding indexes, but will it recover data? Or is it lost?

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.