# Cannot allocate because allocation is not permitted to any of the nodes **URL:** \ **Category:** Elasticsearch\ **Created:** [June 28, 2017, 11:59am UTC](https://discuss.elastic.co/t/cannot-allocate-because-allocation-is-not-permitted-to-any-of-the-nodes/91096 "2017-06-28T11:59:06Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![kabali12345](https://avatars.discourse-cdn.com/v4/letter/k/6a8cbe/32.png) [@kabali12345](https://discuss.elastic.co/u/kabali12345)\ **Post date:** [June 28, 2017, 11:59am UTC](https://discuss.elastic.co/t/cannot-allocate-because-allocation-is-not-permitted-to-any-of-the-nodes/91096/1 "2017-06-28T11:59:06Z")
Hi all i have a 5 node cluster where 3 nodes are data nodes, 1 is master and 1 is client node on this cluster i have created a snapshot for particular index and when i tried to restore that snapshot on same cluster then cluster turns into RED status with unassigned shards = 10 here is the response that get with ```auto GET /_cluster/allocation/explain ``` ```auto { "index": "company", "shard": 2, "primary": false, "current_state": "unassigned", "unassigned_info": { "reason": "NEW_INDEX_RESTORED", "at": "2017-06-28T11:51:00.521Z", "details": "restore_source[sgpl_backup2/snapshot_2]", "last_allocation_status": "no_attempt" }, "can_allocate": "no", "allocate_explanation": "cannot allocate because allocation is not permitted to any of the nodes", "node_allocation_decisions": [ { "node_id": "Uu67NQqXQVaT1maJ49apng", "node_name": "node72", "transport_address": "192.168.1.72:9300", "node_attributes": { "ml.enabled": "true" }, "node_decision": "no", "deciders": [ { "decider": "replica_after_primary_active", "decision": "NO", "explanation": "primary shard for this replica is not yet active" }, { "decider": "throttling", "decision": "NO", "explanation": "primary shard for this replica is not yet active" } ] }, { "node_id": "XEgaW0CvQt-02VEczOmHoA", "node_name": "node71", "transport_address": "192.168.1.71:9300", "node_attributes": { "ml.enabled": "true" }, "node_decision": "no", "deciders": [ { "decider": "replica_after_primary_active", "decision": "NO", "explanation": "primary shard for this replica is not yet active" }, { "decider": "throttling", "decision": "NO", "explanation": "primary shard for this replica is not yet active" } ] }, { "node_id": "mHR3zxDqQCqDvGNBxUF2tw", "node_name": "node73", "transport_address": "192.168.1.73:9300", "node_attributes": { "ml.enabled": "true" }, "node_decision": "no", "deciders": [ { "decider": "replica_after_primary_active", "decision": "NO", "explanation": "primary shard for this replica is not yet active" }, { "decider": "throttling", "decision": "NO", "explanation": "primary shard for this replica is not yet active" } ] } ] } ``` how can i resolve this error? Thank you. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [June 28, 2017, 12:18pm UTC](https://discuss.elastic.co/t/cannot-allocate-because-allocation-is-not-permitted-to-any-of-the-nodes/91096/2 "2017-06-28T12:18:52Z")
Is there anything in the logs? Are you low on disk space? ---
**Author:** ![kabali12345](https://avatars.discourse-cdn.com/v4/letter/k/6a8cbe/32.png) [@kabali12345](https://discuss.elastic.co/u/kabali12345)\ **Post date:** [June 28, 2017, 12:22pm UTC](https://discuss.elastic.co/t/cannot-allocate-because-allocation-is-not-permitted-to-any-of-the-nodes/91096/3 "2017-06-28T12:22:54Z")
This log is from one of the data node also getting same log on master node also ```auto [2017-06-28T21:24:09,023][WARN][o.e.c.a.s.ShardStateAction] [node74] [my_index][0] received shard failed for shard id [[my_index][0]], allocation id [lavnJrtuTLu7FdYGRcPWPw], primary term [0], message [failed recovery], failure [RecoveryFailedException[[my_index][0]: Recovery failed on {node73}{mHR3zxDqQCqDvGNBxUF2tw}{BwILk21wR4GlJaWDy-DHFg}{192.168.1.73}{192.168.1.73:9300}{ml.enabled=true}]; nested: IndexShardRecoveryException[failed recovery]; nested: IndexShardRestoreFailedException[restore failed]; nested: IndexShardRestoreFailedException[failed to restore snapshot [snapshot1/NPgjpNbaSPqIKEFpRw0ihg]]; nested: SnapshotException[[sgpl:snapshot1/NPgjpNbaSPqIKEFpRw0ihg] failed to read shard snapshot file for [my_index][0]]; nested: NoSuchFileException[/home/sgpl/elasticsearch_backup/indices/my_index/0/snap-NPgjpNbaSPqIKEFpRw0ihg.dat]; ] org.elasticsearch.indices.recovery.RecoveryFailedException: [my_index][0]: Recovery failed on {node73}{mHR3zxDqQCqDvGNBxUF2tw}{BwILk21wR4GlJaWDy-DHFg}{192.168.1.73}{192.168.1.73:9300}{ml.enabled=true} at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$2(IndexShard.java:1515) ~[elasticsearch-5.4.0.jar:5.4.0] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:569) ~[elasticsearch-5.4.0.jar:5.4.0] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[?:1.8.0_65] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[?:1.8.0_65] at java.lang.Thread.run(Thread.java:745) [?:1.8.0_65] Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:299) ~[elasticsearch-5.4.0.jar:5.4.0] at org.elasticsearch.index.shard.StoreRecovery.recoverFromRepository(StoreRecovery.java:232) ~[elasticsearch-5.4.0.jar:5.4.0] at org.elasticsearch.index.shard.IndexShard.restoreFromRepository(IndexShard.java:1247) ~[elasticsearch-5.4.0.jar:5.4.0] at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$2(IndexShard.java:1511) ~[elasticsearch-5.4.0.jar:5.4.0] ... 4 more Caused by: org.elasticsearch.index.snapshots.IndexShardRestoreFailedException: restore failed at org.elasticsearch.index.shard.StoreRecovery.restore(StoreRecovery.java:405) ~[elasticsearch-5.4.0.jar:5.4.0] at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromRepository$4(StoreRecovery.java:234) ~[elasticsearch-5.4.0.jar:5.4.0] at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:257) ~[elasticsearch-5.4.0.jar:5.4.0] at org.elasticsearch.index.shard.StoreRecovery.recoverFromRepository(StoreRecovery.java:232) ~[elasticsearch-5.4.0.jar:5.4.0] at org.elasticsearch.index.shard.IndexShard.restoreFromRepository(IndexShard.java:1247) ~[elasticsearch-5.4.0.jar:5.4.0] at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$2(IndexShard.java:1511) ~[elasticsearch-5.4.0.jar:5.4.0] ... 4 more Caused by: org.elasticsearch.index.snapshots.IndexShardRestoreFailedException: failed to restore snapshot [snapshot1/NPgjpNbaSPqIKEFpRw0ihg] at org.elasticsearch.repositories.blobstore.BlobStoreRepository.restoreShard(BlobStoreRepository.java:987) ~[elasticsearch-5.4.0.jar:5.4.0] at org.elasticsearch.index.shard.StoreRecovery.restore(StoreRecovery.java:400) ~[elasticsearch-5.4.0.jar:5.4.0] at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromRepository$4(StoreRecovery.java:234) ~[elasticsearch-5.4.0.jar:5.4.0] at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:257) ~[elasticsearch-5.4.0.jar:5.4.0] at org.elasticsearch.index.shard.StoreRecovery.recoverFromRepository(StoreRecovery.java:232) ~[elasticsearch-5.4.0.jar:5.4.0] at org.elasticsearch.index.shard.IndexShard.restoreFromRepository(IndexShard.java:1247) ~[elasticsearch-5.4.0.jar:5.4.0] at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$2(IndexShard.java:1511) ~[elasticsearch-5.4.0.jar:5.4.0] ... 4 more Caused by: org.elasticsearch.snapshots.SnapshotException: [sgpl:snapshot1/NPgjpNbaSPqIKEFpRw0ihg] failed to read shard snapshot file for [my_index][0] at org.elasticsearch.repositories.blobstore.BlobStoreRepository$Context.loadSnapshot(BlobStoreRepository.java:1114) ~[elasticsearch-5.4.0.jar:5.4.0] at org.elasticsearch.repositories.blobstore.BlobStoreRepository$RestoreContext.restore(BlobStoreRepository.java:1580) ~[elasticsearch-5.4.0.jar:5.4.0] at org.elasticsearch.repositories.blobstore.BlobStoreRepository.restoreShard(BlobStoreRepository.java:985) ~[elasticsearch-5.4.0.jar:5.4.0] at org.elasticsearch.index.shard.StoreRecovery.restore(StoreRecovery.java:400) ~[elasticsearch-5.4.0.jar:5.4.0] at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromRepository$4(StoreRecovery.java:234) ~[elasticsearch-5.4.0.jar:5.4.0] at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:257) ~[elasticsearch-5.4.0.jar:5.4.0] at org.elasticsearch.index.shard.StoreRecovery.recoverFromRepository(StoreRecovery.java:232) ~[elasticsearch-5.4.0.jar:5.4.0] at org.elasticsearch.index.shard.IndexShard.restoreFromRepository(IndexShard.java:1247) ~[elasticsearch-5.4.0.jar:5.4.0] at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$2(IndexShard.java:1511) ~[elasticsearch-5.4.0.jar:5.4.0] ... 4 more Caused by: java.nio.file.NoSuchFileException: /home/sgpl/elasticsearch_backup/indices/my_index/0/snap-NPgjpNbaSPqIKEFpRw0ihg.dat at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86) ~[?:?] at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) ~[?:?] at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) ~[?:?] at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:214) ~[?:?] at java.nio.file.Files.newByteChannel(Files.java:361) ~[?:1.8.0_65] at java.nio.file.Files.newByteChannel(Files.java:407) ~[?:1.8.0_65] at java.nio.file.spi.FileSystemProvider.newInputStream(FileSystemProvider.java:384) ~[?:1.8.0_65] at java.nio.file.Files.newInputStream(Files.java:152) ~[?:1.8.0_65] at org.elasticsearch.common.blobstore.fs.FsBlobContainer.readBlob(FsBlobContainer.java:118) ~[elasticsearch-5.4.0.jar:5.4.0] at org.elasticsearch.repositories.blobstore.ChecksumBlobStoreFormat.readBlob(ChecksumBlobStoreFormat.java:103) ~[elasticsearch-5.4.0.jar:5.4.0] at org.elasticsearch.repositories.blobstore.BlobStoreFormat.read(BlobStoreFormat.java:89) ~[elasticsearch-5.4.0.jar:5.4.0] at org.elasticsearch.repositories.blobstore.BlobStoreRepository$Context.loadSnapshot(BlobStoreRepository.java:1112) ~[elasticsearch-5.4.0.jar:5.4.0] at org.elasticsearch.repositories.blobstore.BlobStoreRepository$RestoreContext.restore(BlobStoreRepository.java:1580) ~[elasticsearch-5.4.0.jar:5.4.0] at ``` ---
**Author:** ![kabali12345](https://avatars.discourse-cdn.com/v4/letter/k/6a8cbe/32.png) [@kabali12345](https://discuss.elastic.co/u/kabali12345)\ **Post date:** [June 28, 2017, 1:41pm UTC](https://discuss.elastic.co/t/cannot-allocate-because-allocation-is-not-permitted-to-any-of-the-nodes/91096/4 "2017-06-28T13:41:15Z")
Also and sufficient disk space ```auto [sgpl@elasticsearch_datanode3 ~]$ df -h Filesystem Size Used Avail Use% Mounted on /dev/mapper/centos-root 101G 9.4G 92G 10% / devtmpfs 5.8G 0 5.8G 0% /dev tmpfs 5.8G 84K 5.8G 1% /dev/shm tmpfs 5.8G 8.9M 5.8G 1% /run tmpfs 5.8G 0 5.8G 0% /sys/fs/cgroup tmpfs 1.2G 16K 1.2G 1% /run/user/42 tmpfs 1.2G 0 1.2G 0% /run/user/1000 ``` ---
**Author:** ![kabali12345](https://avatars.discourse-cdn.com/v4/letter/k/6a8cbe/32.png) [@kabali12345](https://discuss.elastic.co/u/kabali12345)\ **Post date:** [June 28, 2017, 4:07pm UTC](https://discuss.elastic.co/t/cannot-allocate-because-allocation-is-not-permitted-to-any-of-the-nodes/91096/5 "2017-06-28T16:07:32Z")
i have not find the below file on any of the node at that particular location but this file exists on master node in some other location ```auto Caused by: java.nio.file.NoSuchFileException: /home/sgpl/elasticsearch_backup/indices/my_index/0/snap-NPgjpNbaSPqIKEFpRw0ihg.dat ``` ---
**Author:** ![kabali12345](https://avatars.discourse-cdn.com/v4/letter/k/6a8cbe/32.png) [@kabali12345](https://discuss.elastic.co/u/kabali12345)\ **Post date:** [June 28, 2017, 4:22pm UTC](https://discuss.elastic.co/t/cannot-allocate-because-allocation-is-not-permitted-to-any-of-the-nodes/91096/6 "2017-06-28T16:22:18Z")
And when i creates snapshot i am getting following files in master node ```auto [sgpl@elasticsearch_masternode elasticsearch_backup]$ ls index-0 index.latest indices meta-ACuNmZi_ReGvcsAHkF6yzw.dat snap-ACuNmZi_ReGvcsAHkF6yzw.dat ``` and following files in all data nodes ```auto [sgpl@elasticsearch_datanode2 elasticsearch_backup]$ ls indices ``` ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 26, 2017, 4:22pm UTC](https://discuss.elastic.co/t/cannot-allocate-because-allocation-is-not-permitted-to-any-of-the-nodes/91096/7 "2017-07-26T16:22:38Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.