Ok, I have following problems with allocating shards:
GET /_cluster/allocation/explain
{
"index": "index_with_red_status",
"shard": 0,
"primary": true
}
Response:
{
"index" : "index_with_red_status",
"shard" : 0,
"primary" : true,
"current_state" : "initializing",
"unassigned_info" : {
"reason" : "ALLOCATION_FAILED",
"at" : "2020-07-09T10:03:43.306Z",
"failed_allocation_attempts" : 1,
"details" : "failed shard on node [ymRg964ASrmkefI3uWbNrQ]: failed recovery, failure RecoveryFailedException[[index_with_red_status][0]: Recovery failed on {elasticsearch-data-3}{ymRg964ASrmkefI3uWbNrQ}{k53mojOORkqH4G7KUidQaw}{192.168.1.58}{192.168.1.58:9300}]; nested: IndexShardRecoveryException[failed recovery]; nested: IndexShardRestoreFailedException[restore failed]; nested: IndexShardRestoreFailedException[failed to restore snapshot [es-snapshot/ZiOqBRXCTsKZ9nMywzWC4Q]]; nested: IndexShardRestoreFailedException[Failed to recover index]; nested: IOException[Input/output error]; ",
"last_allocation_status" : "awaiting_info"
},
"current_node" : {
"id" : "Uq-XbuBaR7Glzakj47ikOQ",
"name" : "elasticsearch-data-7",
"transport_address" : "192.168.1.206:9300"
},
"explanation" : "the shard is in the process of initializing on node [elasticsearch-data-7], wait until initialization has completed"
}
It happens to random indices during snapshot restore. They are fixed after retry(ies), but could you please tell me what is the possible purpose of the failure? There are no problems with connections between the nodes, all the ES docker containers are up and running, without any restarts.