I keep getting the following exception in Elastic Log on trying to restore a snapshot using cloud_aws plugin from s3 repository:
[WARN ][cluster.action.shard ] [Landslide] [index_name][0] received shard failed for target shard [[index_name][0], node[U2w_femBQYO3f5TuOI5daw], [P], v[109], restoring[elasticsearch:backup.backup_name.16-08-01-0444], s[INITIALIZING], a[id=gJBMpmcVT6G132-h1ONGgw], unassigned_info[[reason=ALLOCATION_FAILED], at[2016-09-01T13:01:53.147Z], details[failed to create shard, failure ElasticsearchException[failed to create shard]; nested: LockObtainFailedException[Can't lock shard [index_name][0], timed out after 5000ms]; ]]], indexUUID [7quZdjJqRRmhzr7WBXqlgQ], message [failed to create shard], failure [ElasticsearchException[failed to create shard]; nested: LockObtainFailedException[Can't lock shard [index_name][0], timed out after 5000ms]; ]
[index_name][[index_name][0]] ElasticsearchException[failed to create shard]; nested: LockObtainFailedException[Can't lock shard [index_name][0], timed out after 5000ms];
at org.elasticsearch.index.IndexService.createShard(IndexService.java:389)
at org.elasticsearch.indices.cluster.IndicesClusterStateService.applyInitializingShard(IndicesClusterStateService.java:601)
at org.elasticsearch.indices.cluster.IndicesClusterStateService.applyNewOrUpdatedShards(IndicesClusterStateService.java:501)
at org.elasticsearch.indices.cluster.IndicesClusterStateService.clusterChanged(IndicesClusterStateService.java:166)
at org.elasticsearch.cluster.service.InternalClusterService.runTasksForExecutor(InternalClusterService.java:610)
at org.elasticsearch.cluster.service.InternalClusterService$UpdateTask.run(InternalClusterService.java:772)
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:231)
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:194)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.lucene.store.LockObtainFailedException: Can't lock shard [index_name][0], timed out after 5000ms
at org.elasticsearch.env.NodeEnvironment$InternalShardLock.acquire(NodeEnvironment.java:609)
at org.elasticsearch.env.NodeEnvironment.shardLock(NodeEnvironment.java:537)
at org.elasticsearch.index.IndexService.createShard(IndexService.java:306)
... 10 more
I am running ES version 2.4.0 on my laptop (Mac El-Capitan). There's no other operation going on on elastic server. Also note that the restore operation does complete inspite of the above exception and sometimes it doesn't. Any suggestions?