I have two Elasticsearch nodes on server A and B .Each node is storing the
indices on two SANs (Storage area network) mounted on each server.
I booted up another server C with a single elastic search node on it. And
then, unmounted one of the SAN on the server A and mounted it on server C.
I restarted the whole cluster again with the new placement of the SANs. I
assumed that the cluster will find the manually moved shards and it will
initialize them accordingly . But this isn't the case. I am getting the
below warning messages in the logs.
sending failed shard for [2013-12-26.06:00][0], node[VD7c5WgxSaaTZ2_6yRgcLQ
], [P], s[INITIALIZING], reason [Failed to start shard, message [
IndexShardGatewayRecoveryException[[2013-12-26.06:00][0] shard allocated for
local recovery (post api), should exist, but doesn't, current files:[......
The above warn message is pretty straightforward. It just means that the
local recovery for a particular index failed due to SAN relocation, which
is valid.
But I want the cluster to search for the shards on other nodes too. But
somehow the moved shards are not getting located by the cluster.
Is there a way to initialize these manually moved shards correctly?
Also, can anyone elaborate on what meta information that is stored in the
Elasticsearch index which is used by the local node and also the whole
cluster during recovery ?
Oh , btw I am using Elasticsearch version - 0.90.5
They won't be detected as the cluster metadata will have the shards
allocated to node A, not node C.
The metadata needs to be updated as well as you point out, but I don't
believe there is a way to manually do that.
On 22 January 2014 11:39, hrishikesh prabhune hruship@gmail.com wrote:
Hello All,
I have two Elasticsearch nodes on server A and B .Each node is storing the
indices on two SANs (Storage area network) mounted on each server.
I booted up another server C with a single Elasticsearch node on it. And
then, unmounted one of the SAN on the server A and mounted it on server C.
I restarted the whole cluster again with the new placement of the SANs. I
assumed that the cluster will find the manually moved shards and it will
initialize them accordingly . But this isn't the case. I am getting the
below warning messages in the logs.
sending failed shard for [2013-12-26.06:00][0], node[
VD7c5WgxSaaTZ2_6yRgcLQ], [P], s[INITIALIZING], reason [Failed to start
shard, message [IndexShardGatewayRecoveryException[[2013-12-26.06:00][0]shard allocated
for local recovery (post api), should exist, but doesn't, current
files:[......
The above warn message is pretty straightforward. It just means that the
local recovery for a particular index failed due to SAN relocation, which
is valid.
But I want the cluster to search for the shards on other nodes too. But
somehow the moved shards are not getting located by the cluster.
Is there a way to initialize these manually moved shards correctly?
Also, can anyone elaborate on what meta information that is stored in the
Elasticsearch index which is used by the local node and also the whole
cluster during recovery ?
Oh , btw I am using Elasticsearch version - 0.90.5
Thanks for a precise and prompt reply!
Will updating to Elasticsearch 0.90.10 help me in recovering these shards?
This bug Multi data path config can cause a shard to be perceived as corrupted · Issue #4674 · elastic/elasticsearch · GitHub
potentially talks about a similar case that I am facing. And this bug
is
fixed in 0.90.10.
Also , the above issue talks about segments.gen file . I deleted the
segments.gen file from a particular index. But still the shards were not
allocated correctly.
They won't be detected as the cluster metadata will have the shards
allocated to node A, not node C.
The metadata needs to be updated as well as you point out, but I don't
believe there is a way to manually do that.
On 22 January 2014 11:39, hrishikesh prabhune hruship@gmail.com wrote:
Hello All,
I have two Elasticsearch nodes on server A and B .Each node is storing
the indices on two SANs (Storage area network) mounted on each server.
I booted up another server C with a single Elasticsearch node on it. And
then, unmounted one of the SAN on the server A and mounted it on server C.
I restarted the whole cluster again with the new placement of the SANs. I
assumed that the cluster will find the manually moved shards and it will
initialize them accordingly . But this isn't the case. I am getting the
below warning messages in the logs.
sending failed shard for [2013-12-26.06:00][0], node[
VD7c5WgxSaaTZ2_6yRgcLQ], [P], s[INITIALIZING], reason [Failed to start
shard, message [IndexShardGatewayRecoveryException[[2013-12-26.06:00][0]shard allocated
for local recovery (post api), should exist, but doesn't, current
files:[......
The above warn message is pretty straightforward. It just means that the
local recovery for a particular index failed due to SAN relocation, which
is valid.
But I want the cluster to search for the shards on other nodes too. But
somehow the moved shards are not getting located by the cluster.
Is there a way to initialize these manually moved shards correctly?
Also, can anyone elaborate on what meta information that is stored in
the Elasticsearch index which is used by the local node and also the whole
cluster during recovery ?
Oh , btw I am using Elasticsearch version - 0.90.5
Thanks in advance!
--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.