Cannot allocate because allocation is not permitted to any of the nodes

Hi all
i have a 5 node cluster where 3 nodes are data nodes, 1 is master and 1 is client node
on this cluster i have created a snapshot for particular index and when i tried to restore that snapshot on same cluster then cluster turns into RED status with unassigned shards = 10
here is the response that get with

GET /_cluster/allocation/explain
{
  "index": "company",
  "shard": 2,
  "primary": false,
  "current_state": "unassigned",
  "unassigned_info": {
    "reason": "NEW_INDEX_RESTORED",
    "at": "2017-06-28T11:51:00.521Z",
    "details": "restore_source[sgpl_backup2/snapshot_2]",
    "last_allocation_status": "no_attempt"
  },
  "can_allocate": "no",
  "allocate_explanation": "cannot allocate because allocation is not permitted to any of the nodes",
  "node_allocation_decisions": [
    {
      "node_id": "Uu67NQqXQVaT1maJ49apng",
      "node_name": "node72",
      "transport_address": "192.168.1.72:9300",
      "node_attributes": {
        "ml.enabled": "true"
      },
      "node_decision": "no",
      "deciders": [
        {
          "decider": "replica_after_primary_active",
          "decision": "NO",
          "explanation": "primary shard for this replica is not yet active"
        },
        {
          "decider": "throttling",
          "decision": "NO",
          "explanation": "primary shard for this replica is not yet active"
        }
      ]
    },
    {
      "node_id": "XEgaW0CvQt-02VEczOmHoA",
      "node_name": "node71",
      "transport_address": "192.168.1.71:9300",
      "node_attributes": {
        "ml.enabled": "true"
      },
      "node_decision": "no",
      "deciders": [
        {
          "decider": "replica_after_primary_active",
          "decision": "NO",
          "explanation": "primary shard for this replica is not yet active"
        },
        {
          "decider": "throttling",
          "decision": "NO",
          "explanation": "primary shard for this replica is not yet active"
        }
      ]
    },
    {
      "node_id": "mHR3zxDqQCqDvGNBxUF2tw",
      "node_name": "node73",
      "transport_address": "192.168.1.73:9300",
      "node_attributes": {
        "ml.enabled": "true"
      },
      "node_decision": "no",
      "deciders": [
        {
          "decider": "replica_after_primary_active",
          "decision": "NO",
          "explanation": "primary shard for this replica is not yet active"
        },
        {
          "decider": "throttling",
          "decision": "NO",
          "explanation": "primary shard for this replica is not yet active"
        }
      ]
    }
  ]
}

how can i resolve this error?

Thank you.

Is there anything in the logs? Are you low on disk space?

This log is from one of the data node also getting same log on master node also

[2017-06-28T21:24:09,023][WARN ][o.e.c.a.s.ShardStateAction] [node74] [my_index][0] received shard failed for shard id [[my_index][0]], allocation id [lavnJrtuTLu7FdYGRcPWPw], primary term [0], message [failed recovery], failure [RecoveryFailedException[[my_index][0]: Recovery failed on {node73}{mHR3zxDqQCqDvGNBxUF2tw}{BwILk21wR4GlJaWDy-DHFg}{192.168.1.73}{192.168.1.73:9300}{ml.enabled=true}]; nested: IndexShardRecoveryException[failed recovery]; nested: IndexShardRestoreFailedException[restore failed]; nested: IndexShardRestoreFailedException[failed to restore snapshot [snapshot1/NPgjpNbaSPqIKEFpRw0ihg]]; nested: SnapshotException[[sgpl:snapshot1/NPgjpNbaSPqIKEFpRw0ihg] failed to read shard snapshot file for [my_index][0]]; nested: NoSuchFileException[/home/sgpl/elasticsearch_backup/indices/my_index/0/snap-NPgjpNbaSPqIKEFpRw0ihg.dat]; ]
org.elasticsearch.indices.recovery.RecoveryFailedException: [my_index][0]: Recovery failed on {node73}{mHR3zxDqQCqDvGNBxUF2tw}{BwILk21wR4GlJaWDy-DHFg}{192.168.1.73}{192.168.1.73:9300}{ml.enabled=true}
	at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$2(IndexShard.java:1515) ~[elasticsearch-5.4.0.jar:5.4.0]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:569) ~[elasticsearch-5.4.0.jar:5.4.0]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[?:1.8.0_65]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[?:1.8.0_65]
	at java.lang.Thread.run(Thread.java:745) [?:1.8.0_65]
Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery
	at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:299) ~[elasticsearch-5.4.0.jar:5.4.0]
	at org.elasticsearch.index.shard.StoreRecovery.recoverFromRepository(StoreRecovery.java:232) ~[elasticsearch-5.4.0.jar:5.4.0]
	at org.elasticsearch.index.shard.IndexShard.restoreFromRepository(IndexShard.java:1247) ~[elasticsearch-5.4.0.jar:5.4.0]
	at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$2(IndexShard.java:1511) ~[elasticsearch-5.4.0.jar:5.4.0]
	... 4 more
Caused by: org.elasticsearch.index.snapshots.IndexShardRestoreFailedException: restore failed
	at org.elasticsearch.index.shard.StoreRecovery.restore(StoreRecovery.java:405) ~[elasticsearch-5.4.0.jar:5.4.0]
	at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromRepository$4(StoreRecovery.java:234) ~[elasticsearch-5.4.0.jar:5.4.0]
	at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:257) ~[elasticsearch-5.4.0.jar:5.4.0]
	at org.elasticsearch.index.shard.StoreRecovery.recoverFromRepository(StoreRecovery.java:232) ~[elasticsearch-5.4.0.jar:5.4.0]
	at org.elasticsearch.index.shard.IndexShard.restoreFromRepository(IndexShard.java:1247) ~[elasticsearch-5.4.0.jar:5.4.0]
	at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$2(IndexShard.java:1511) ~[elasticsearch-5.4.0.jar:5.4.0]
	... 4 more
Caused by: org.elasticsearch.index.snapshots.IndexShardRestoreFailedException: failed to restore snapshot [snapshot1/NPgjpNbaSPqIKEFpRw0ihg]
	at org.elasticsearch.repositories.blobstore.BlobStoreRepository.restoreShard(BlobStoreRepository.java:987) ~[elasticsearch-5.4.0.jar:5.4.0]
	at org.elasticsearch.index.shard.StoreRecovery.restore(StoreRecovery.java:400) ~[elasticsearch-5.4.0.jar:5.4.0]
	at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromRepository$4(StoreRecovery.java:234) ~[elasticsearch-5.4.0.jar:5.4.0]
	at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:257) ~[elasticsearch-5.4.0.jar:5.4.0]
	at org.elasticsearch.index.shard.StoreRecovery.recoverFromRepository(StoreRecovery.java:232) ~[elasticsearch-5.4.0.jar:5.4.0]
	at org.elasticsearch.index.shard.IndexShard.restoreFromRepository(IndexShard.java:1247) ~[elasticsearch-5.4.0.jar:5.4.0]
	at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$2(IndexShard.java:1511) ~[elasticsearch-5.4.0.jar:5.4.0]
	... 4 more
Caused by: org.elasticsearch.snapshots.SnapshotException: [sgpl:snapshot1/NPgjpNbaSPqIKEFpRw0ihg] failed to read shard snapshot file for [my_index][0]
	at org.elasticsearch.repositories.blobstore.BlobStoreRepository$Context.loadSnapshot(BlobStoreRepository.java:1114) ~[elasticsearch-5.4.0.jar:5.4.0]
	at org.elasticsearch.repositories.blobstore.BlobStoreRepository$RestoreContext.restore(BlobStoreRepository.java:1580) ~[elasticsearch-5.4.0.jar:5.4.0]
	at org.elasticsearch.repositories.blobstore.BlobStoreRepository.restoreShard(BlobStoreRepository.java:985) ~[elasticsearch-5.4.0.jar:5.4.0]
	at org.elasticsearch.index.shard.StoreRecovery.restore(StoreRecovery.java:400) ~[elasticsearch-5.4.0.jar:5.4.0]
	at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromRepository$4(StoreRecovery.java:234) ~[elasticsearch-5.4.0.jar:5.4.0]
	at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:257) ~[elasticsearch-5.4.0.jar:5.4.0]
	at org.elasticsearch.index.shard.StoreRecovery.recoverFromRepository(StoreRecovery.java:232) ~[elasticsearch-5.4.0.jar:5.4.0]
	at org.elasticsearch.index.shard.IndexShard.restoreFromRepository(IndexShard.java:1247) ~[elasticsearch-5.4.0.jar:5.4.0]
	at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$2(IndexShard.java:1511) ~[elasticsearch-5.4.0.jar:5.4.0]
	... 4 more
Caused by: java.nio.file.NoSuchFileException: /home/sgpl/elasticsearch_backup/indices/my_index/0/snap-NPgjpNbaSPqIKEFpRw0ihg.dat
	at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86) ~[?:?]
	at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) ~[?:?]
	at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) ~[?:?]
	at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:214) ~[?:?]
	at java.nio.file.Files.newByteChannel(Files.java:361) ~[?:1.8.0_65]
	at java.nio.file.Files.newByteChannel(Files.java:407) ~[?:1.8.0_65]
	at java.nio.file.spi.FileSystemProvider.newInputStream(FileSystemProvider.java:384) ~[?:1.8.0_65]
	at java.nio.file.Files.newInputStream(Files.java:152) ~[?:1.8.0_65]
	at org.elasticsearch.common.blobstore.fs.FsBlobContainer.readBlob(FsBlobContainer.java:118) ~[elasticsearch-5.4.0.jar:5.4.0]
	at org.elasticsearch.repositories.blobstore.ChecksumBlobStoreFormat.readBlob(ChecksumBlobStoreFormat.java:103) ~[elasticsearch-5.4.0.jar:5.4.0]
	at org.elasticsearch.repositories.blobstore.BlobStoreFormat.read(BlobStoreFormat.java:89) ~[elasticsearch-5.4.0.jar:5.4.0]
	at org.elasticsearch.repositories.blobstore.BlobStoreRepository$Context.loadSnapshot(BlobStoreRepository.java:1112) ~[elasticsearch-5.4.0.jar:5.4.0]
	at org.elasticsearch.repositories.blobstore.BlobStoreRepository$RestoreContext.restore(BlobStoreRepository.java:1580) ~[elasticsearch-5.4.0.jar:5.4.0]
	at 

Also and sufficient disk space

[sgpl@elasticsearch_datanode3 ~]$ df -h
Filesystem               Size  Used Avail Use% Mounted on
/dev/mapper/centos-root  101G  9.4G   92G  10% /
devtmpfs                 5.8G     0  5.8G   0% /dev
tmpfs                    5.8G   84K  5.8G   1% /dev/shm
tmpfs                    5.8G  8.9M  5.8G   1% /run
tmpfs                    5.8G     0  5.8G   0% /sys/fs/cgroup
tmpfs                    1.2G   16K  1.2G   1% /run/user/42
tmpfs                    1.2G     0  1.2G   0% /run/user/1000

i have not find the below file on any of the node at that particular location but this file exists on master node in some other location

Caused by: java.nio.file.NoSuchFileException: /home/sgpl/elasticsearch_backup/indices/my_index/0/snap-NPgjpNbaSPqIKEFpRw0ihg.dat

And when i creates snapshot i am getting following files in master node

[sgpl@elasticsearch_masternode elasticsearch_backup]$ ls
index-0  index.latest  indices  meta-ACuNmZi_ReGvcsAHkF6yzw.dat  snap-ACuNmZi_ReGvcsAHkF6yzw.dat

and following files in all data nodes

[sgpl@elasticsearch_datanode2 elasticsearch_backup]$ ls
indices