I'm actively trying to move from Elasticsearch 1.6 to 2.3.
I was trying to snapshot my indices to prepare the migration but my backup system failed at the wrong time and my ES cluster had to be stopped due to non-stop timeouts.
After a full cluster restart, some of my indices are red and I can't seem to find them again:
All of them are primary shards.
"status" : "red",
"timed_out" : false,
"number_of_nodes" : 5,
"number_of_data_nodes" : 5,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 54,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0
Shard allocation is correctly enabled on all the machines
I've tried to POST a reroute but without success.
When trying the _recovery option on the bad indices, the missing shards do not even appear on the returned JSON.
Is there something I'm missing ?
UPDATE: The data is definitely on the nodes, I can see the folders with the id of the missing
shards. Is there an API call to verify data on disks to force check the shards availability ?