# Encountering Index shard gateway recovery exception while manually moving the shards across nodes **URL:** \ **Category:** Elasticsearch\ **Created:** [January 22, 2014, 12:39am UTC](https://discuss.elastic.co/t/encountering-index-shard-gateway-recovery-exception-while-manually-moving-the-shards-across-nodes/15351 "2014-01-22T00:39:47Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![hrishikesh\_prabhune](https://avatars.discourse-cdn.com/v4/letter/h/6f9a4e/32.png) [@hrishikesh\_prabhune](https://discuss.elastic.co/u/hrishikesh_prabhune)\ **Post date:** [January 22, 2014, 12:39am UTC](https://discuss.elastic.co/t/encountering-index-shard-gateway-recovery-exception-while-manually-moving-the-shards-across-nodes/15351/1 "2014-01-22T00:39:47Z")
Hello All, I have two Elasticsearch nodes on server A and B .Each node is storing the indices on two SANs (Storage area network) mounted on each server. I booted up another server C with a single elastic search node on it. And then, unmounted one of the SAN on the server A and mounted it on server C. I restarted the whole cluster again with the new placement of the SANs. I assumed that the cluster will find the manually moved shards and it will initialize them accordingly . But this isn't the case. I am getting the below warning messages in the logs. sending failed shard for [2013-12-26.06:00][0], node[VD7c5WgxSaaTZ2\_6yRgcLQ ], [P], s[INITIALIZING], reason [Failed to start shard, message [ IndexShardGatewayRecoveryException[[2013-12-26.06:00][0] shard allocated for local recovery (post api), should exist, but doesn't, current files:[...... The above warn message is pretty straightforward. It just means that the local recovery for a particular index failed due to SAN relocation, which is valid. But I want the cluster to search for the shards on other nodes too. But somehow the moved shards are not getting located by the cluster. Is there a way to initialize these manually moved shards correctly? Also, can anyone elaborate on what meta information that is stored in the Elasticsearch index which is used by the local node and also the whole cluster during recovery ? Oh , btw I am using Elasticsearch version - 0.90.5 Thanks in advance! -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/cfccfc5a-6521-4a39-8d28-201daa5b7484%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/cfccfc5a-6521-4a39-8d28-201daa5b7484%40googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [January 22, 2014, 12:53am UTC](https://discuss.elastic.co/t/encountering-index-shard-gateway-recovery-exception-while-manually-moving-the-shards-across-nodes/15351/2 "2014-01-22T00:53:42Z")
They won't be detected as the cluster metadata will have the shards allocated to node A, not node C. The metadata needs to be updated as well as you point out, but I don't believe there is a way to manually do that. Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: [markw@campaignmonitor.com](mailto:markw@campaignmonitor.com) web: [www.campaignmonitor.com](http://www.campaignmonitor.com) On 22 January 2014 11:39, hrishikesh prabhune [hruship@gmail.com](mailto:hruship@gmail.com) wrote: > Hello All, > > I have two Elasticsearch nodes on server A and B .Each node is storing the > indices on two SANs (Storage area network) mounted on each server. > I booted up another server C with a single Elasticsearch node on it. And > then, unmounted one of the SAN on the server A and mounted it on server C. > I restarted the whole cluster again with the new placement of the SANs. I > assumed that the cluster will find the manually moved shards and it will > initialize them accordingly . But this isn't the case. I am getting the > below warning messages in the logs. > sending failed shard for [2013-12-26.06:00][0], node[ > VD7c5WgxSaaTZ2\_6yRgcLQ], [P], s[INITIALIZING], reason [Failed to start > shard, message [IndexShardGatewayRecoveryException[[2013-12-26.06:00][0]shard allocated > for local recovery (post api), should exist, but doesn't, current > files:[...... > The above warn message is pretty straightforward. It just means that the > local recovery for a particular index failed due to SAN relocation, which > is valid. > But I want the cluster to search for the shards on other nodes too. But > somehow the moved shards are not getting located by the cluster. > Is there a way to initialize these manually moved shards correctly? > Also, can anyone elaborate on what meta information that is stored in the > Elasticsearch index which is used by the local node and also the whole > cluster during recovery ? > Oh , btw I am using Elasticsearch version - 0.90.5 > > Thanks in advance! > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/cfccfc5a-6521-4a39-8d28-201daa5b7484%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/cfccfc5a-6521-4a39-8d28-201daa5b7484%40googlegroups.com) > . > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAEM624aOsMEsa-BUF33LJ6vO\_m8w-0ytiAuPBKr%3DKuDsn2qdRw%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEM624aOsMEsa-BUF33LJ6vO_m8w-0ytiAuPBKr%3DKuDsn2qdRw%40mail.gmail.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![hrishikesh\_prabhune](https://avatars.discourse-cdn.com/v4/letter/h/6f9a4e/32.png) [@hrishikesh\_prabhune](https://discuss.elastic.co/u/hrishikesh_prabhune)\ **Post date:** [January 22, 2014, 9:07am UTC](https://discuss.elastic.co/t/encountering-index-shard-gateway-recovery-exception-while-manually-moving-the-shards-across-nodes/15351/3 "2014-01-22T09:07:48Z")
Thanks for a precise and prompt reply! Will updating to Elasticsearch 0.90.10 help me in recovering these shards? This bug [Multi data path config can cause a shard to be perceived as corrupted · Issue #4674 · elastic/elasticsearch · GitHub](https://github.com/elasticsearch/elasticsearch/issues/4674) potentially talks about a similar case that I am facing. And this bug is fixed in 0.90.10. Also , the above issue talks about segments.gen file . I deleted the segments.gen file from a particular index. But still the shards were not allocated correctly. Thanks! On Tue, Jan 21, 2014 at 4:53 PM, Mark Walkom [markw@campaignmonitor.com](mailto:markw@campaignmonitor.com)wrote: > They won't be detected as the cluster metadata will have the shards > allocated to node A, not node C. > The metadata needs to be updated as well as you point out, but I don't > believe there is a way to manually do that. > > Regards, > Mark Walkom > > Infrastructure Engineer > Campaign Monitor > email: [markw@campaignmonitor.com](mailto:markw@campaignmonitor.com) > web: [www.campaignmonitor.com](http://www.campaignmonitor.com) > > On 22 January 2014 11:39, hrishikesh prabhune [hruship@gmail.com](mailto:hruship@gmail.com) wrote: > > > Hello All, > > > > I have two Elasticsearch nodes on server A and B .Each node is storing > > the indices on two SANs (Storage area network) mounted on each server. > > I booted up another server C with a single Elasticsearch node on it. And > > then, unmounted one of the SAN on the server A and mounted it on server C. > > I restarted the whole cluster again with the new placement of the SANs. I > > assumed that the cluster will find the manually moved shards and it will > > initialize them accordingly . But this isn't the case. I am getting the > > below warning messages in the logs. > > sending failed shard for [2013-12-26.06:00][0], node[ > > VD7c5WgxSaaTZ2\_6yRgcLQ], [P], s[INITIALIZING], reason [Failed to start > > shard, message [IndexShardGatewayRecoveryException[[2013-12-26.06:00][0]shard allocated > > for local recovery (post api), should exist, but doesn't, current > > files:[...... > > The above warn message is pretty straightforward. It just means that the > > local recovery for a particular index failed due to SAN relocation, which > > is valid. > > But I want the cluster to search for the shards on other nodes too. But > > somehow the moved shards are not getting located by the cluster. > > Is there a way to initialize these manually moved shards correctly? > > Also, can anyone elaborate on what meta information that is stored in > > the Elasticsearch index which is used by the local node and also the whole > > cluster during recovery ? > > Oh , btw I am using Elasticsearch version - 0.90.5 > > > > Thanks in advance! > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/cfccfc5a-6521-4a39-8d28-201daa5b7484%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/cfccfc5a-6521-4a39-8d28-201daa5b7484%40googlegroups.com) > > . > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > -- > You received this message because you are subscribed to a topic in the > Google Groups "elasticsearch" group. > To unsubscribe from this topic, visit > [https://groups.google.com/d/topic/elasticsearch/RDfCv3JKxaA/unsubscribe](https://groups.google.com/d/topic/elasticsearch/RDfCv3JKxaA/unsubscribe). > To unsubscribe from this group and all its topics, send an email to > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/CAEM624aOsMEsa-BUF33LJ6vO\_m8w-0ytiAuPBKr%3DKuDsn2qdRw%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEM624aOsMEsa-BUF33LJ6vO_m8w-0ytiAuPBKr%3DKuDsn2qdRw%40mail.gmail.com) > . > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CA%2BSd%2B\_fpVZVdTMA0%3DvxHPmZzPA18DZmDgJAVaOSgYnQx4B29Ug%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CA%2BSd%2B_fpVZVdTMA0%3DvxHPmZzPA18DZmDgJAVaOSgYnQx4B29Ug%40mail.gmail.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 1:55am UTC](https://discuss.elastic.co/t/encountering-index-shard-gateway-recovery-exception-while-manually-moving-the-shards-across-nodes/15351/4 "2017-07-06T01:55:26Z")