Cannot allocate because allocation is not permitted to any of the nodes

Hi all
i have a 5 node cluster where 3 nodes are data nodes, 1 is master and 1 is client node
on this cluster i have created a snapshot for particular index and when i tried to restore that snapshot on same cluster then cluster turns into RED status with unassigned shards = 10
here is the response that get with

GET /_cluster/allocation/explain
{
  "index": "company",
  "shard": 2,
  "primary": false,
  "current_state": "unassigned",
  "unassigned_info": {
    "reason": "NEW_INDEX_RESTORED",
    "at": "2017-06-28T11:51:00.521Z",
    "details": "restore_source[sgpl_backup2/snapshot_2]",
    "last_allocation_status": "no_attempt"
  },
  "can_allocate": "no",
  "allocate_explanation": "cannot allocate because allocation is not permitted to any of the nodes",
  "node_allocation_decisions": [
    {
      "node_id": "Uu67NQqXQVaT1maJ49apng",
      "node_name": "node72",
      "transport_address": "192.168.1.72:9300",
      "node_attributes": {
        "ml.enabled": "true"
      },
      "node_decision": "no",
      "deciders": [
        {
          "decider": "replica_after_primary_active",
          "decision": "NO",
          "explanation": "primary shard for this replica is not yet active"
        },
        {
          "decider": "throttling",
          "decision": "NO",
          "explanation": "primary shard for this replica is not yet active"
        }
      ]
    },
    {
      "node_id": "XEgaW0CvQt-02VEczOmHoA",
      "node_name": "node71",
      "transport_address": "192.168.1.71:9300",
      "node_attributes": {
        "ml.enabled": "true"
      },
      "node_decision": "no",
      "deciders": [
        {
          "decider": "replica_after_primary_active",
          "decision": "NO",
          "explanation": "primary shard for this replica is not yet active"
        },
        {
          "decider": "throttling",
          "decision": "NO",
          "explanation": "primary shard for this replica is not yet active"
        }
      ]
    },
    {
      "node_id": "mHR3zxDqQCqDvGNBxUF2tw",
      "node_name": "node73",
      "transport_address": "192.168.1.73:9300",
      "node_attributes": {
        "ml.enabled": "true"
      },
      "node_decision": "no",
      "deciders": [
        {
          "decider": "replica_after_primary_active",
          "decision": "NO",
          "explanation": "primary shard for this replica is not yet active"
        },
        {
          "decider": "throttling",
          "decision": "NO",
          "explanation": "primary shard for this replica is not yet active"
        }
      ]
    }
  ]
}

how can i resolve this error?

Thank you.

Is there anything in the logs? Are you low on disk space?

1 Like

This log is from one of the data node also getting same log on master node also

[2017-06-28T21:24:09,023][WARN ][o.e.c.a.s.ShardStateAction] [node74] [my_index][0] received shard failed for shard id [[my_index][0]], allocation id [lavnJrtuTLu7FdYGRcPWPw], primary term [0], message [failed recovery], failure [RecoveryFailedException[[my_index][0]: Recovery failed on {node73}{mHR3zxDqQCqDvGNBxUF2tw}{BwILk21wR4GlJaWDy-DHFg}{192.168.1.73}{192.168.1.73:9300}{ml.enabled=true}]; nested: IndexShardRecoveryException[failed recovery]; nested: IndexShardRestoreFailedException[restore failed]; nested: IndexShardRestoreFailedException[failed to restore snapshot [snapshot1/NPgjpNbaSPqIKEFpRw0ihg]]; nested: SnapshotException[[sgpl:snapshot1/NPgjpNbaSPqIKEFpRw0ihg] failed to read shard snapshot file for [my_index][0]]; nested: NoSuchFileException[/home/sgpl/elasticsearch_backup/indices/my_index/0/snap-NPgjpNbaSPqIKEFpRw0ihg.dat]; ]
org.elasticsearch.indices.recovery.RecoveryFailedException: [my_index][0]: Recovery failed on {node73}{mHR3zxDqQCqDvGNBxUF2tw}{BwILk21wR4GlJaWDy-DHFg}{192.168.1.73}{192.168.1.73:9300}{ml.enabled=true}
	at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$2(IndexShard.java:1515) ~[elasticsearch-5.4.0.jar:5.4.0]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:569) ~[elasticsearch-5.4.0.jar:5.4.0]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[?:1.8.0_65]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[?:1.8.0_65]
	at java.lang.Thread.run(Thread.java:745) [?:1.8.0_65]
Caused by: org.elasticsearch.index.shard.IndexShardRecoveryException: failed recovery
	at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:299) ~[elasticsearch-5.4.0.jar:5.4.0]
	at org.elasticsearch.index.shard.StoreRecovery.recoverFromRepository(StoreRecovery.java:232) ~[elasticsearch-5.4.0.jar:5.4.0]
	at org.elasticsearch.index.shard.IndexShard.restoreFromRepository(IndexShard.java:1247) ~[elasticsearch-5.4.0.jar:5.4.0]
	at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$2(IndexShard.java:1511) ~[elasticsearch-5.4.0.jar:5.4.0]
	... 4 more
Caused by: org.elasticsearch.index.snapshots.IndexShardRestoreFailedException: restore failed
	at org.elasticsearch.index.shard.StoreRecovery.restore(StoreRecovery.java:405) ~[elasticsearch-5.4.0.jar:5.4.0]
	at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromRepository$4(StoreRecovery.java:234) ~[elasticsearch-5.4.0.jar:5.4.0]
	at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:257) ~[elasticsearch-5.4.0.jar:5.4.0]
	at org.elasticsearch.index.shard.StoreRecovery.recoverFromRepository(StoreRecovery.java:232) ~[elasticsearch-5.4.0.jar:5.4.0]
	at org.elasticsearch.index.shard.IndexShard.restoreFromRepository(IndexShard.java:1247) ~[elasticsearch-5.4.0.jar:5.4.0]
	at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$2(IndexShard.java:1511) ~[elasticsearch-5.4.0.jar:5.4.0]
	... 4 more
Caused by: org.elasticsearch.index.snapshots.IndexShardRestoreFailedException: failed to restore snapshot [snapshot1/NPgjpNbaSPqIKEFpRw0ihg]
	at org.elasticsearch.repositories.blobstore.BlobStoreRepository.restoreShard(BlobStoreRepository.java:987) ~[elasticsearch-5.4.0.jar:5.4.0]
	at org.elasticsearch.index.shard.StoreRecovery.restore(StoreRecovery.java:400) ~[elasticsearch-5.4.0.jar:5.4.0]
	at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromRepository$4(StoreRecovery.java:234) ~[elasticsearch-5.4.0.jar:5.4.0]
	at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:257) ~[elasticsearch-5.4.0.jar:5.4.0]
	at org.elasticsearch.index.shard.StoreRecovery.recoverFromRepository(StoreRecovery.java:232) ~[elasticsearch-5.4.0.jar:5.4.0]
	at org.elasticsearch.index.shard.IndexShard.restoreFromRepository(IndexShard.java:1247) ~[elasticsearch-5.4.0.jar:5.4.0]
	at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$2(IndexShard.java:1511) ~[elasticsearch-5.4.0.jar:5.4.0]
	... 4 more
Caused by: org.elasticsearch.snapshots.SnapshotException: [sgpl:snapshot1/NPgjpNbaSPqIKEFpRw0ihg] failed to read shard snapshot file for [my_index][0]
	at org.elasticsearch.repositories.blobstore.BlobStoreRepository$Context.loadSnapshot(BlobStoreRepository.java:1114) ~[elasticsearch-5.4.0.jar:5.4.0]
	at org.elasticsearch.repositories.blobstore.BlobStoreRepository$RestoreContext.restore(BlobStoreRepository.java:1580) ~[elasticsearch-5.4.0.jar:5.4.0]
	at org.elasticsearch.repositories.blobstore.BlobStoreRepository.restoreShard(BlobStoreRepository.java:985) ~[elasticsearch-5.4.0.jar:5.4.0]
	at org.elasticsearch.index.shard.StoreRecovery.restore(StoreRecovery.java:400) ~[elasticsearch-5.4.0.jar:5.4.0]
	at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromRepository$4(StoreRecovery.java:234) ~[elasticsearch-5.4.0.jar:5.4.0]
	at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:257) ~[elasticsearch-5.4.0.jar:5.4.0]
	at org.elasticsearch.index.shard.StoreRecovery.recoverFromRepository(StoreRecovery.java:232) ~[elasticsearch-5.4.0.jar:5.4.0]
	at org.elasticsearch.index.shard.IndexShard.restoreFromRepository(IndexShard.java:1247) ~[elasticsearch-5.4.0.jar:5.4.0]
	at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$2(IndexShard.java:1511) ~[elasticsearch-5.4.0.jar:5.4.0]
	... 4 more
Caused by: java.nio.file.NoSuchFileException: /home/sgpl/elasticsearch_backup/indices/my_index/0/snap-NPgjpNbaSPqIKEFpRw0ihg.dat
	at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86) ~[?:?]
	at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) ~[?:?]
	at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) ~[?:?]
	at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:214) ~[?:?]
	at java.nio.file.Files.newByteChannel(Files.java:361) ~[?:1.8.0_65]
	at java.nio.file.Files.newByteChannel(Files.java:407) ~[?:1.8.0_65]
	at java.nio.file.spi.FileSystemProvider.newInputStream(FileSystemProvider.java:384) ~[?:1.8.0_65]
	at java.nio.file.Files.newInputStream(Files.java:152) ~[?:1.8.0_65]
	at org.elasticsearch.common.blobstore.fs.FsBlobContainer.readBlob(FsBlobContainer.java:118) ~[elasticsearch-5.4.0.jar:5.4.0]
	at org.elasticsearch.repositories.blobstore.ChecksumBlobStoreFormat.readBlob(ChecksumBlobStoreFormat.java:103) ~[elasticsearch-5.4.0.jar:5.4.0]
	at org.elasticsearch.repositories.blobstore.BlobStoreFormat.read(BlobStoreFormat.java:89) ~[elasticsearch-5.4.0.jar:5.4.0]
	at org.elasticsearch.repositories.blobstore.BlobStoreRepository$Context.loadSnapshot(BlobStoreRepository.java:1112) ~[elasticsearch-5.4.0.jar:5.4.0]
	at org.elasticsearch.repositories.blobstore.BlobStoreRepository$RestoreContext.restore(BlobStoreRepository.java:1580) ~[elasticsearch-5.4.0.jar:5.4.0]
	at 

Also and sufficient disk space

[sgpl@elasticsearch_datanode3 ~]$ df -h
Filesystem               Size  Used Avail Use% Mounted on
/dev/mapper/centos-root  101G  9.4G   92G  10% /
devtmpfs                 5.8G     0  5.8G   0% /dev
tmpfs                    5.8G   84K  5.8G   1% /dev/shm
tmpfs                    5.8G  8.9M  5.8G   1% /run
tmpfs                    5.8G     0  5.8G   0% /sys/fs/cgroup
tmpfs                    1.2G   16K  1.2G   1% /run/user/42
tmpfs                    1.2G     0  1.2G   0% /run/user/1000

i have not find the below file on any of the node at that particular location but this file exists on master node in some other location

Caused by: java.nio.file.NoSuchFileException: /home/sgpl/elasticsearch_backup/indices/my_index/0/snap-NPgjpNbaSPqIKEFpRw0ihg.dat

And when i creates snapshot i am getting following files in master node

[sgpl@elasticsearch_masternode elasticsearch_backup]$ ls
index-0  index.latest  indices  meta-ACuNmZi_ReGvcsAHkF6yzw.dat  snap-ACuNmZi_ReGvcsAHkF6yzw.dat

and following files in all data nodes

[sgpl@elasticsearch_datanode2 elasticsearch_backup]$ ls
indices

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.