It seems until the shards have not successfully recovered, ES service re-tries the restore operation on that shard. Sometimes two or more shards get placed in a single disk, this will be an issue if you have setup the cluster in such a way that a single disk mount can accommodate only one shard. eg total shards per disk: 6 and each disk is 200 GB so each disk can hold shard < 200GB but the instance itself can hold a total of 6 shards. Suppose 2 shards get placed in a single disk you will run out of space. To solve the issue just move the one of the shard folder from the disk with two of them to the disk which has none. Restart the es service in the node only. Please check the _recovery API for the progress. The shards should be in INDEX stage.
Restoring failed shards from Snapshot due to Caused by: java.io.IOException: No space left on device
Sudan_Khadgi (Sudan Khadgi) #1