Encountering Index shard gateway recovery exception while manually moving the shards across nodes


(hrishikesh prabhune) #1

Hello All,

I have two Elasticsearch nodes on server A and B .Each node is storing the
indices on two SANs (Storage area network) mounted on each server.
I booted up another server C with a single elastic search node on it. And
then, unmounted one of the SAN on the server A and mounted it on server C.
I restarted the whole cluster again with the new placement of the SANs. I
assumed that the cluster will find the manually moved shards and it will
initialize them accordingly . But this isn't the case. I am getting the
below warning messages in the logs.
sending failed shard for [2013-12-26.06:00][0], node[VD7c5WgxSaaTZ2_6yRgcLQ
], [P], s[INITIALIZING], reason [Failed to start shard, message [
IndexShardGatewayRecoveryException[[2013-12-26.06:00][0] shard allocated for
local recovery (post api), should exist, but doesn't, current files:[......
The above warn message is pretty straightforward. It just means that the
local recovery for a particular index failed due to SAN relocation, which
is valid.
But I want the cluster to search for the shards on other nodes too. But
somehow the moved shards are not getting located by the cluster.
Is there a way to initialize these manually moved shards correctly?
Also, can anyone elaborate on what meta information that is stored in the
Elasticsearch index which is used by the local node and also the whole
cluster during recovery ?
Oh , btw I am using Elasticsearch version - 0.90.5

Thanks in advance!

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/cfccfc5a-6521-4a39-8d28-201daa5b7484%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(Mark Walkom) #2

They won't be detected as the cluster metadata will have the shards
allocated to node A, not node C.
The metadata needs to be updated as well as you point out, but I don't
believe there is a way to manually do that.

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: markw@campaignmonitor.com
web: www.campaignmonitor.com

On 22 January 2014 11:39, hrishikesh prabhune hruship@gmail.com wrote:

Hello All,

I have two Elasticsearch nodes on server A and B .Each node is storing the
indices on two SANs (Storage area network) mounted on each server.
I booted up another server C with a single elastic search node on it. And
then, unmounted one of the SAN on the server A and mounted it on server C.
I restarted the whole cluster again with the new placement of the SANs. I
assumed that the cluster will find the manually moved shards and it will
initialize them accordingly . But this isn't the case. I am getting the
below warning messages in the logs.
sending failed shard for [2013-12-26.06:00][0], node[
VD7c5WgxSaaTZ2_6yRgcLQ], [P], s[INITIALIZING], reason [Failed to start
shard, message [IndexShardGatewayRecoveryException[[2013-12-26.06:00][0]shard allocated
for local recovery (post api), should exist, but doesn't, current
files:[......
The above warn message is pretty straightforward. It just means that the
local recovery for a particular index failed due to SAN relocation, which
is valid.
But I want the cluster to search for the shards on other nodes too. But
somehow the moved shards are not getting located by the cluster.
Is there a way to initialize these manually moved shards correctly?
Also, can anyone elaborate on what meta information that is stored in the
Elasticsearch index which is used by the local node and also the whole
cluster during recovery ?
Oh , btw I am using Elasticsearch version - 0.90.5

Thanks in advance!

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/cfccfc5a-6521-4a39-8d28-201daa5b7484%40googlegroups.com
.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEM624aOsMEsa-BUF33LJ6vO_m8w-0ytiAuPBKr%3DKuDsn2qdRw%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.


(hrishikesh prabhune) #3

Thanks for a precise and prompt reply!
Will updating to Elasticsearch 0.90.10 help me in recovering these shards?
This bug https://github.com/elasticsearch/elasticsearch/issues/4674
potentially talks about a similar case that I am facing. And this bug
is
fixed in 0.90.10.
Also , the above issue talks about segments.gen file . I deleted the
segments.gen file from a particular index. But still the shards were not
allocated correctly.

Thanks!

On Tue, Jan 21, 2014 at 4:53 PM, Mark Walkom markw@campaignmonitor.comwrote:

They won't be detected as the cluster metadata will have the shards
allocated to node A, not node C.
The metadata needs to be updated as well as you point out, but I don't
believe there is a way to manually do that.

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: markw@campaignmonitor.com
web: www.campaignmonitor.com

On 22 January 2014 11:39, hrishikesh prabhune hruship@gmail.com wrote:

Hello All,

I have two Elasticsearch nodes on server A and B .Each node is storing
the indices on two SANs (Storage area network) mounted on each server.
I booted up another server C with a single elastic search node on it. And
then, unmounted one of the SAN on the server A and mounted it on server C.
I restarted the whole cluster again with the new placement of the SANs. I
assumed that the cluster will find the manually moved shards and it will
initialize them accordingly . But this isn't the case. I am getting the
below warning messages in the logs.
sending failed shard for [2013-12-26.06:00][0], node[
VD7c5WgxSaaTZ2_6yRgcLQ], [P], s[INITIALIZING], reason [Failed to start
shard, message [IndexShardGatewayRecoveryException[[2013-12-26.06:00][0]shard allocated
for local recovery (post api), should exist, but doesn't, current
files:[......
The above warn message is pretty straightforward. It just means that the
local recovery for a particular index failed due to SAN relocation, which
is valid.
But I want the cluster to search for the shards on other nodes too. But
somehow the moved shards are not getting located by the cluster.
Is there a way to initialize these manually moved shards correctly?
Also, can anyone elaborate on what meta information that is stored in
the Elasticsearch index which is used by the local node and also the whole
cluster during recovery ?
Oh , btw I am using Elasticsearch version - 0.90.5

Thanks in advance!

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.

To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/cfccfc5a-6521-4a39-8d28-201daa5b7484%40googlegroups.com
.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/RDfCv3JKxaA/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAEM624aOsMEsa-BUF33LJ6vO_m8w-0ytiAuPBKr%3DKuDsn2qdRw%40mail.gmail.com
.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CA%2BSd%2B_fpVZVdTMA0%3DvxHPmZzPA18DZmDgJAVaOSgYnQx4B29Ug%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.


(system) #4