# After adding a third node to the cluster, some shards won't relocate **URL:** \ **Category:** Elasticsearch\ **Created:** [February 13, 2014, 6:25pm UTC](https://discuss.elastic.co/t/after-adding-a-third-node-to-the-cluster-some-shards-wont-relocate/15786 "2014-02-13T18:25:22Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![Christer](https://avatars.discourse-cdn.com/v4/letter/c/a5b964/32.png) [@Christer](https://discuss.elastic.co/u/Christer)\ **Post date:** [February 13, 2014, 6:25pm UTC](https://discuss.elastic.co/t/after-adding-a-third-node-to-the-cluster-some-shards-wont-relocate/15786/1 "2014-02-13T18:25:22Z")
Earlier today I added a third node to our cluster. It shares the same version of elasticsearch (0.90.10) and jvm (1.7.0\_13) as the two existing nodes. Now, some hours after I added the node, two shards are still "relocating". The status of the cluster is green though. I'm getting some errors in the log of the node I added: [2014-02-13 19:13:43,572][WARN][cluster.action.shard] [elasticsearch03] [vgd][3] sending failed shard for [vgd][3], node[LgR5cuiCQmSfOTfTl6t1qA], relocating [VuACiBeiToyz7xEZ5RJsxQ], [P], s[INITIALIZING], indexUUID [-5I0LkSET8GXIaOLCpnQUQ], reason [Failed to start shard, message [RecoveryFailedException[[vgd][3]: Recovery failed from [elasticsearch01][VuACiBeiToyz7xEZ5RJsxQ][inet[/10.84.200.129:9300]] into [elasticsearch03][LgR5cuiCQmSfOTfTl6t1qA][inet[/10.84.100.219:9300]]]; nested: RemoteTransportException[[elasticsearch01][inet[/10.84.200.129:9300]][index/shard/recovery/startRecovery]]; nested: RecoveryEngineException[[vgd][3] Phase[2] Execution failed]; nested: ReceiveTimeoutTransportException[[elasticsearch03][inet[/10.84.100.219:9300]][index/shard/recovery/prepareTranslog] request\_id [712931] timed out after [900000ms]]; ]] It says it "timed out", but there is no connection issues between the nodes as far as I can tell. The new node has ~2M docs, whereas node1 and 2 has ~45M (which is the total amount of indexed docs). The new node also uses quite a lot CPU, as it has been doing since it joined the cluster earlier today. Any tips on how to debug this problem any further so I can have a three node cluster up and running? -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/3624b153-3ad5-4a24-8e3d-f189e714c9fd%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/3624b153-3ad5-4a24-8e3d-f189e714c9fd%40googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Christer](https://avatars.discourse-cdn.com/v4/letter/c/a5b964/32.png) [@Christer](https://discuss.elastic.co/u/Christer)\ **Post date:** [February 13, 2014, 6:30pm UTC](https://discuss.elastic.co/t/after-adding-a-third-node-to-the-cluster-some-shards-wont-relocate/15786/2 "2014-02-13T18:30:44Z")
There is quite a lot of errors such as the one I pasted btw. On Thursday, February 13, 2014 7:25:22 PM UTC+1, Christer wrote: > Earlier today I added a third node to our cluster. It shares the same > version of elasticsearch (0.90.10) and jvm (1.7.0\_13) as the two existing > nodes. > > Now, some hours after I added the node, two shards are still "relocating". > The status of the cluster is green though. I'm getting some errors in the > log of the node I added: > > [2014-02-13 19:13:43,572][WARN][cluster.action.shard] > [elasticsearch03] [vgd][3] sending failed shard for [vgd][3], > node[LgR5cuiCQmSfOTfTl6t1qA], relocating [VuACiBeiToyz7xEZ5RJsxQ], [P], > s[INITIALIZING], indexUUID [-5I0LkSET8GXIaOLCpnQUQ], reason [Failed to > start shard, message [RecoveryFailedException[[vgd][3]: Recovery failed > from [elasticsearch01][VuACiBeiToyz7xEZ5RJsxQ][inet[/10.84.200.129:9300]] > into [elasticsearch03][LgR5cuiCQmSfOTfTl6t1qA][inet[/10.84.100.219:9300]]]; > nested: > RemoteTransportException[[elasticsearch01][inet[/10.84.200.129:9300]][index/shard/recovery/startRecovery]]; > nested: RecoveryEngineException[[vgd][3] Phase[2] Execution failed]; > nested: > ReceiveTimeoutTransportException[[elasticsearch03][inet[/10.84.100.219:9300]][index/shard/recovery/prepareTranslog] > request\_id [712931] timed out after [900000ms]]; ]] > > It says it "timed out", but there is no connection issues between the > nodes as far as I can tell. The new node has ~2M docs, whereas node1 and 2 > has ~45M (which is the total amount of indexed docs). The new node also > uses quite a lot CPU, as it has been doing since it joined the cluster > earlier today. > > Any tips on how to debug this problem any further so I can have a three > node cluster up and running? -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/8e4eb4f5-9f73-4d21-819a-743ce0e0ffb6%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/8e4eb4f5-9f73-4d21-819a-743ce0e0ffb6%40googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Christer](https://avatars.discourse-cdn.com/v4/letter/c/a5b964/32.png) [@Christer](https://discuss.elastic.co/u/Christer)\ **Post date:** [February 13, 2014, 6:45pm UTC](https://discuss.elastic.co/t/after-adding-a-third-node-to-the-cluster-some-shards-wont-relocate/15786/3 "2014-02-13T18:45:38Z")
And on elasticsearch01 (the node referred to in the error message) I'm seeing a whole lot of these: [2014-02-13 19:13:42,391][WARN][transport] [elasticsearch01] Received response for a request that has timed out, sent [1138869ms] ago, timed out [238869ms] ago, action [index/shard/recovery/prepareTranslog], node [[elasticsearch03][LgR5cuiCQmSfOTfTl6t1qA][inet[/10.84.100.219:9300]]], id [702105] [2014-02-13 19:13:43,573][WARN][cluster.action.shard] [elasticsearch01] [vgd][3] received shard failed for [vgd][3], node[LgR5cuiCQmSfOTfTl6t1qA], relocating [VuACiBeiToyz7xEZ5RJsxQ], [P], s[INITIALIZING], indexUUID [-5I0LkSET8GXIaOLCpnQUQ], reason [Failed to start shard, message [RecoveryFailedException[[vgd][3]: Recovery failed from [elasticsearch01][VuACiBeiToyz7xEZ5RJsxQ][inet[/10.84.200.129:9300]] into [elasticsearch03][LgR5cuiCQmSfOTfTl6t1qA][inet[/10.84.100.219:9300]]]; nested: RemoteTransportException[[elasticsearch01][inet[/10.84.200.129:9300]][index/shard/recovery/startRecovery]]; nested: RecoveryEngineException[[vgd][3] Phase[2] Execution failed]; nested: ReceiveTimeoutTransportException[[elasticsearch03][inet[/10.84.100.219:9300]][index/shard/recovery/prepareTranslog] request\_id [712931] timed out after [900000ms]]; ]] On Thursday, February 13, 2014 7:25:22 PM UTC+1, Christer wrote: > Earlier today I added a third node to our cluster. It shares the same > version of elasticsearch (0.90.10) and jvm (1.7.0\_13) as the two existing > nodes. > > Now, some hours after I added the node, two shards are still "relocating". > The status of the cluster is green though. I'm getting some errors in the > log of the node I added: > > [2014-02-13 19:13:43,572][WARN][cluster.action.shard] > [elasticsearch03] [vgd][3] sending failed shard for [vgd][3], > node[LgR5cuiCQmSfOTfTl6t1qA], relocating [VuACiBeiToyz7xEZ5RJsxQ], [P], > s[INITIALIZING], indexUUID [-5I0LkSET8GXIaOLCpnQUQ], reason [Failed to > start shard, message [RecoveryFailedException[[vgd][3]: Recovery failed > from [elasticsearch01][VuACiBeiToyz7xEZ5RJsxQ][inet[/10.84.200.129:9300]] > into [elasticsearch03][LgR5cuiCQmSfOTfTl6t1qA][inet[/10.84.100.219:9300]]]; > nested: > RemoteTransportException[[elasticsearch01][inet[/10.84.200.129:9300]][index/shard/recovery/startRecovery]]; > nested: RecoveryEngineException[[vgd][3] Phase[2] Execution failed]; > nested: > ReceiveTimeoutTransportException[[elasticsearch03][inet[/10.84.100.219:9300]][index/shard/recovery/prepareTranslog] > request\_id [712931] timed out after [900000ms]]; ]] > > It says it "timed out", but there is no connection issues between the > nodes as far as I can tell. The new node has ~2M docs, whereas node1 and 2 > has ~45M (which is the total amount of indexed docs). The new node also > uses quite a lot CPU, as it has been doing since it joined the cluster > earlier today. > > Any tips on how to debug this problem any further so I can have a three > node cluster up and running? -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/3a946189-29cf-4e3b-b066-fa28decb36d8%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/3a946189-29cf-4e3b-b066-fa28decb36d8%40googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Christer](https://avatars.discourse-cdn.com/v4/letter/c/a5b964/32.png) [@Christer](https://discuss.elastic.co/u/Christer)\ **Post date:** [February 14, 2014, 9:35am UTC](https://discuss.elastic.co/t/after-adding-a-third-node-to-the-cluster-some-shards-wont-relocate/15786/4 "2014-02-14T09:35:05Z")
After I disabled the index.shard.check\_on\_startup: true option from the node, it managed to join the cluster, and all shards have since been successfully relocated. On Thursday, February 13, 2014 7:25:22 PM UTC+1, Christer wrote: > Earlier today I added a third node to our cluster. It shares the same > version of elasticsearch (0.90.10) and jvm (1.7.0\_13) as the two existing > nodes. > > Now, some hours after I added the node, two shards are still "relocating". > The status of the cluster is green though. I'm getting some errors in the > log of the node I added: > > [2014-02-13 19:13:43,572][WARN][cluster.action.shard] > [elasticsearch03] [vgd][3] sending failed shard for [vgd][3], > node[LgR5cuiCQmSfOTfTl6t1qA], relocating [VuACiBeiToyz7xEZ5RJsxQ], [P], > s[INITIALIZING], indexUUID [-5I0LkSET8GXIaOLCpnQUQ], reason [Failed to > start shard, message [RecoveryFailedException[[vgd][3]: Recovery failed > from [elasticsearch01][VuACiBeiToyz7xEZ5RJsxQ][inet[/10.84.200.129:9300]] > into [elasticsearch03][LgR5cuiCQmSfOTfTl6t1qA][inet[/10.84.100.219:9300]]]; > nested: > RemoteTransportException[[elasticsearch01][inet[/10.84.200.129:9300]][index/shard/recovery/startRecovery]]; > nested: RecoveryEngineException[[vgd][3] Phase[2] Execution failed]; > nested: > ReceiveTimeoutTransportException[[elasticsearch03][inet[/10.84.100.219:9300]][index/shard/recovery/prepareTranslog] > request\_id [712931] timed out after [900000ms]]; ]] > > It says it "timed out", but there is no connection issues between the > nodes as far as I can tell. The new node has ~2M docs, whereas node1 and 2 > has ~45M (which is the total amount of indexed docs). The new node also > uses quite a lot CPU, as it has been doing since it joined the cluster > earlier today. > > Any tips on how to debug this problem any further so I can have a three > node cluster up and running? -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/5ef76d28-8240-4773-91b8-9b1bcf45dfad%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/5ef76d28-8240-4773-91b8-9b1bcf45dfad%40googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 1:50am UTC](https://discuss.elastic.co/t/after-adding-a-third-node-to-the-cluster-some-shards-wont-relocate/15786/5 "2017-07-06T01:50:14Z")