How to Fix Unallocated Shards?

[Siva_Sai]

now stuck at 75.2% curl -XGET -u elastic:changeme 'localhost:9200/_cat/health?pretty' 1484915420 12:30:20 elasticsearch red 5 5 5656 2828 0 0 1870 0 - 75.2%

I can't fix it - tried to restore many times - gets up to 67.2% what do we do? restore from many days back? this shows many unallocated shards curl -XGET 'localhost:9200/_cat/shards?pretty' -u elastic:

[Christian_Dahlqvist]

Please provide some more details. Which Elasticsearch version are you on? What its the size and specification of the cluster? How many indices/shards do you have and what is the total data volume?

What is the output of the cat nodes API?

[Siva_Sai]

Elastic search version : 5.1.1
Size and specification of the cluster: size: 6 specification: 2TB disk 16GB RAM CPUS : 6
no.of shards: primary shards:2000+
active shards: 5000+
unassigned shards: 2000+

[Christian_Dahlqvist]

What is the output of the _cat/nodes API?

[Siva_Sai]

[dakrone (Lee Hinman)]

You could also try the cluster allocation explain API:

https://www.elastic.co/guide/en/elasticsearch/reference/5.1/cluster-allocation-explain.html

Determine which shard is unallocated and then do:

GET /_cluster/allocation/explain
{
  "index": "my_index",
  "shard": 0,
  "primary": false
}

Replace my_index with your index name, 0 with the shard number that is
unassigned and false with true if the unassigned shard you wish to explain
is a primary shard.

This should give you a nice output for why the shard is unassigned.

[Siva_Sai]

"the shard cannot be assigned because allocation deciders return a NO decision.
primary shard for this replica is not yet active

unassigned:
Reason: "cluster recovered"

[Siva_Sai]

Please find the output attachment

[dakrone (Lee Hinman)]

It looks like the primary copy of you should is stale, therefore there is no
valid copy of the data that can be promoted to primary.

If you are curious, if you read more about stale shards here:
https://www.elastic.co/guide/en/elasticsearch/reference/5.1/docs-replication.html#_failure_handling

By any chance, did this cluster experience a split brain due to two different
masters? Was minimum_master_nodes configured correctly the entire time?