Failed to start shard

Elastic Stack Elasticsearch
1

Hello,

Our master is constantly logging this error, about twice per second....

I saw similar error mentioned in the release notes for a previous verion, this cluster is running 1.0.1. Is it the same issue? What work arounds do we have?

[2014-07-22 23:50:28,533][WARN ][cluster.action.shard ] [rap-es2] [de_2007][0] received shard failed for [de_2007][0], node[EtDIFKT7Qi-nAuvG47tGYQ], [P], s[INITIALIZING], indexUUID [yLrHy5k7RqO6rLnl_b4kww], reason [Failed to start shard, message [IndexShardGatewayRecoveryException[[de_2007][0] failed to fetch index version after copying it over]; nested: IndexShardGatewayRecoveryException[[de_2007][0] shard allocated for local recovery (post api), should exist, but doesn't, current files: []]; nested: IndexNotFoundException[no segments* file found in store(least_used[rate_limited(mmapfs(/data/elasticsearch/data/Odine2/nodes/0/indices/de_2007/0/index), type=MERGE, rate=20.0)]): files: []]; ]]

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/fd9dfaef-7179-4abf-bb60-b8a93982d76a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

2

I went ahead and upgraded to elasticsearch-1.2.2-1.noarch, but the same
errors continues after cluster start.

I notice that the node inquestion is the master, and is supposed to be a
non data node. it should not be starting any shards, right?

Thanks a lot,
Mohamed.

On Tue, Jul 22, 2014 at 11:55 PM, Mohamed Lrhazi ml623@georgetown.edu
wrote:

Hello,

Our master is constantly logging this error, about twice per second....

I saw similar error mentioned in the release notes for a previous verion, this cluster is running 1.0.1. Is it the same issue? What work arounds do we have?

[2014-07-22 23:50:28,533][WARN ][cluster.action.shard ] [rap-es2] [de_2007][0] received shard failed for [de_2007][0], node[EtDIFKT7Qi-nAuvG47tGYQ], [P], s[INITIALIZING], indexUUID [yLrHy5k7RqO6rLnl_b4kww], reason [Failed to start shard, message [IndexShardGatewayRecoveryException[[de_2007][0] failed to fetch index version after copying it over]; nested: IndexShardGatewayRecoveryException[[de_2007][0] shard allocated for local recovery (post api), should exist, but doesn't, current files: ]; nested: IndexNotFoundException[no segments* file found in store(least_used[rate_limited(mmapfs(/data/elasticsearch/data/Odine2/nodes/0/indices/de_2007/0/index), type=MERGE, rate=20.0)]): files: ]; ]]

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/OVc_0Kqm2yk/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/fd9dfaef-7179-4abf-bb60-b8a93982d76a%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/fd9dfaef-7179-4abf-bb60-b8a93982d76a%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEU_gmfNi6nowVSM1FMxWn6XKZkk_szcy1%3DCVZ%3Dz%2BZWoZC%3D8HA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

3

Hi Mohamed,

Did you find a solution to this problem?

On Wednesday, July 23, 2014 10:06:16 AM UTC+5:30, Mohamed Lrhazi wrote:

I went ahead and upgraded to elasticsearch-1.2.2-1.noarch, but the same
errors continues after cluster start.

I notice that the node inquestion is the master, and is supposed to be a
non data node. it should not be starting any shards, right?

Thanks a lot,
Mohamed.

On Tue, Jul 22, 2014 at 11:55 PM, Mohamed Lrhazi <ml...@georgetown.edu
<javascript:>> wrote:

Hello,

Our master is constantly logging this error, about twice per second....

I saw similar error mentioned in the release notes for a previous verion, this cluster is running 1.0.1. Is it the same issue? What work arounds do we have?

[2014-07-22 23:50:28,533][WARN ][cluster.action.shard ] [rap-es2] [de_2007][0] received shard failed for [de_2007][0], node[EtDIFKT7Qi-nAuvG47tGYQ], [P], s[INITIALIZING], indexUUID [yLrHy5k7RqO6rLnl_b4kww], reason [Failed to start shard, message [IndexShardGatewayRecoveryException[[de_2007][0] failed to fetch index version after copying it over]; nested: IndexShardGatewayRecoveryException[[de_2007][0] shard allocated for local recovery (post api), should exist, but doesn't, current files: ]; nested: IndexNotFoundException[no segments* file found in store(least_used[rate_limited(mmapfs(/data/elasticsearch/data/Odine2/nodes/0/indices/de_2007/0/index), type=MERGE, rate=20.0)]): files: ]; ]]

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/OVc_0Kqm2yk/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/fd9dfaef-7179-4abf-bb60-b8a93982d76a%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/fd9dfaef-7179-4abf-bb60-b8a93982d76a%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/52f5a6c8-847c-4294-a2eb-308eb9aa3fe7%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

4

Sorry dont recall! but I find this in an email sent on the same thread
think:

For me the change that seemed to have worked, I keep my fingers crossed,
is to add node.master=False to all nodes except one... black magic!

Other things we did involve: Adding more nodes and deleting indexes with
zero documents.

HTH,
Mohamed.

On Mon, Oct 27, 2014 at 8:12 AM, Anirudh Perugu anirudh@unbxd.com wrote:

Hi Mohamed,

Did you find a solution to this problem?

On Wednesday, July 23, 2014 10:06:16 AM UTC+5:30, Mohamed Lrhazi wrote:

I went ahead and upgraded to elasticsearch-1.2.2-1.noarch, but the same
errors continues after cluster start.

I notice that the node inquestion is the master, and is supposed to be a
non data node. it should not be starting any shards, right?

Thanks a lot,
Mohamed.

On Tue, Jul 22, 2014 at 11:55 PM, Mohamed Lrhazi ml...@georgetown.edu
wrote:

Hello,

Our master is constantly logging this error, about twice per second....

I saw similar error mentioned in the release notes for a previous verion, this cluster is running 1.0.1. Is it the same issue? What work arounds do we have?

[2014-07-22 23:50:28,533][WARN ][cluster.action.shard ] [rap-es2] [de_2007][0] received shard failed for [de_2007][0], node[EtDIFKT7Qi-nAuvG47tGYQ], [P], s[INITIALIZING], indexUUID [yLrHy5k7RqO6rLnl_b4kww], reason [Failed to start shard, message [IndexShardGatewayRecoveryException[[de_2007][0] failed to fetch index version after copying it over]; nested: IndexShardGatewayRecoveryException[[de_2007][0] shard allocated for local recovery (post api), should exist, but doesn't, current files: ]; nested: IndexNotFoundException[no segments* file found in store(least_used[rate_limited(mmapfs(/data/elasticsearch/data/Odine2/nodes/0/indices/de_2007/0/index), type=MERGE, rate=20.0)]): files: ]; ]]

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit https://groups.google.com/d/
topic/elasticsearch/OVc_0Kqm2yk/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/fd9dfaef-7179-4abf-bb60-b8a93982d76a%
40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/fd9dfaef-7179-4abf-bb60-b8a93982d76a%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/OVc_0Kqm2yk/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/52f5a6c8-847c-4294-a2eb-308eb9aa3fe7%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/52f5a6c8-847c-4294-a2eb-308eb9aa3fe7%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEU_gmdtPD8Cxg%2BJmqJhtZRgMv%3DQ8SNo7qds1GT%2BOdegkGKQ-w%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.