# Unassigned replica shards after cluster recovery **URL:** \ **Category:** Elasticsearch\ **Created:** [February 29, 2016, 7:58am UTC](https://discuss.elastic.co/t/unassigned-replica-shards-after-cluster-recovery/42976 "2016-02-29T07:58:35Z")\ **Posts on this page:** 3\ **Page:** 1
**Author:** ![psfu](https://avatars.discourse-cdn.com/v4/letter/p/dc4da7/32.png) [@psfu](https://discuss.elastic.co/u/psfu)\ **Post date:** [February 29, 2016, 7:58am UTC](https://discuss.elastic.co/t/unassigned-replica-shards-after-cluster-recovery/42976/1 "2016-02-29T07:58:35Z")
We are using ES 1.5.2 with 5 x 128G memory server. Java 1.7.0\_67. The cluster now can't assigned replica shards, after a cluster recovery(this had occured before, mostly caused by a long time GC ). The error info as below: > [2016-02-29 13:08:01,111][WARN][indices.cluster] [10.209.240.31] [[idx2-20160228][1]] marking and sending shard failed due to [failed recovery] > org.elasticsearch.indices.recovery.RecoveryFailedException: [idx2-20160228][1]: Recovery failed from [10.209.240.11][8bDsBWlsSYGQvltZQgFrdA][CDM3E01-209240011][inet[/10.209.240.11:11300]] into [10.209.240.31][HljL5MgpQtKWCEMX0nSCSA][CDM3E02-209240031][inet[/10.209.240.31:11300]] > at org.elasticsearch.indices.recovery.RecoveryTarget.doRecovery(RecoveryTarget.java:274) > at org.elasticsearch.indices.recovery.RecoveryTarget.access$700(RecoveryTarget.java:69) > at org.elasticsearch.indices.recovery.RecoveryTarget$RecoveryRunner.doRun(RecoveryTarget.java:550) > at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:36) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > Caused by: org.elasticsearch.transport.RemoteTransportException: [10.209.240.11][inet[/10.209.240.11:11300]][internal:index/shard/recovery/start\_recovery] > Caused by: org.elasticsearch.index.engine.RecoveryEngineException: [idx2-20160228][1] Phase[1] Execution failed > at org.elasticsearch.index.engine.InternalEngine.recover(InternalEngine.java:842) > at org.elasticsearch.index.shard.IndexShard.recover(IndexShard.java:699) > at org.elasticsearch.indices.recovery.RecoverySource.recover(RecoverySource.java:125) > at org.elasticsearch.indices.recovery.RecoverySource.access$200(RecoverySource.java:49) > at org.elasticsearch.indices.recovery.RecoverySource$StartRecoveryTransportRequestHandler.messageReceived(RecoverySource.java:146) > at org.elasticsearch.indices.recovery.RecoverySource$StartRecoveryTransportRequestHandler.messageReceived(RecoverySource.java:132) > at org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.doRun(MessageChannelHandler.java:277) > at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:36) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > Caused by: org.elasticsearch.indices.recovery.RecoverFilesRecoveryException: [idx2-20160228][1] Failed to transfer [118] files with total size of [3.1gb] > at org.elasticsearch.indices.recovery.RecoverySourceHandler.phase1(RecoverySourceHandler.java:413) > at org.elasticsearch.index.engine.InternalEngine.recover(InternalEngine.java:837) > ... 10 more > Caused by: org.elasticsearch.transport.ReceiveTimeoutTransportException: [10.209.240.31][inet[/10.209.240.31:11300]][internal:index/shard/recovery/file\_chunk] request\_id [649263751] timed out after [60ms] > at org.elasticsearch.transport.TransportService$TimeoutHandler.run(TransportService.java:529) > ... 3 more > Suppressed: org.elasticsearch.transport.ReceiveTimeoutTransportException: [10.209.240.31][inet[/10.209.240.31:11300]][internal:index/shard/recovery/file\_chunk] request\_id [649263739] timed out after [60ms] > ... 4 more > Suppressed: org.elasticsearch.transport.ReceiveTimeoutTransportException: [10.209.240.31][inet[/10.209.240.31:11300]][internal:index/shard/recovery/file\_chunk] request\_id [649263731] timed out after [60ms] > ... 4 more > Suppressed: org.elasticsearch.transport.ReceiveTimeoutTransportException: [10.209.240.31][inet[/10.209.240.31:11300]][internal:index/shard/recovery/file\_chunk] request\_id [649263727] timed out after [60ms] How should we do for this? Thanks! ---
**Author:** ![psfu](https://avatars.discourse-cdn.com/v4/letter/p/dc4da7/32.png) [@psfu](https://discuss.elastic.co/u/psfu)\ **Post date:** [February 29, 2016, 8:06am UTC](https://discuss.elastic.co/t/unassigned-replica-shards-after-cluster-recovery/42976/2 "2016-02-29T08:06:21Z")
recovering to ![](https://us1.discourse-cdn.com/elastic/original/2X/1/12261c8c7885be5078c8713bea7d1af964f98ffe.png) recovering from ![](https://us1.discourse-cdn.com/elastic/original/2X/d/d63181c9beada0e417bcaeb9711c08a1bf14697e.png) ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 5, 2017, 11:12pm UTC](https://discuss.elastic.co/t/unassigned-replica-shards-after-cluster-recovery/42976/3 "2017-07-05T23:12:57Z")