# ES 0.20.5 stuck in RED after node loss OR how do I configured to avoid problems? **URL:** \ **Category:** Elasticsearch\ **Created:** [March 11, 2013, 1:07pm UTC](https://discuss.elastic.co/t/es-0-20-5-stuck-in-red-after-node-loss-or-how-do-i-configured-to-avoid-problems/11096 "2013-03-11T13:07:03Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![Andy\_Wick](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andy_wick/32/44017_2.png) [@Andy\_Wick](https://discuss.elastic.co/u/Andy_Wick)\ **Post date:** [March 11, 2013, 1:07pm UTC](https://discuss.elastic.co/t/es-0-20-5-stuck-in-red-after-node-loss-or-how-do-i-configured-to-avoid-problems/11096/1 "2013-03-11T13:07:03Z")
Recently upgraded from 0.19.12 to 0.20.5. A node went down over the weekend in my 30 node cluster. I use ES as a DB so there were constant writes while the node was down. Restarted the node and went RED. [2013-03-10 15:28:22,209][WARN][indices.cluster] [moloches-m13b] [stats][0] failed to start shard org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [stats][0] shard allocated for local recovery (post api), should exists, but doesn't at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:122) at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:177) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) What is the correct way to recover? I have full replication on for this index, and it is a tiny index (8 documents for total store size of 10k) { "stats" : { "settings" : { "index.number\_of\_replicas" : "29", "index.auto\_expand\_replicas" : "0-all", "index.number\_of\_shards" : "1", "index.version.created" : "200599" } } } My desire is that ES should just take care of this, I have 29 other copies why do I need to do anything? 🙂 In a dream world ES would delete the broken index and copy from somewhere else. What am I missing? Thanks, Andy -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Clinton\_Gormley](https://avatars.discourse-cdn.com/v4/letter/c/50afbb/32.png) [@Clinton\_Gormley](https://discuss.elastic.co/u/Clinton_Gormley)\ **Post date:** [March 11, 2013, 3:14pm UTC](https://discuss.elastic.co/t/es-0-20-5-stuck-in-red-after-node-loss-or-how-do-i-configured-to-avoid-problems/11096/2 "2013-03-11T15:14:02Z")
Hiya On Mon, 2013-03-11 at 06:07 -0700, Andy Wick wrote: > Recently upgraded from 0.19.12 to 0.20.5. A node went down over the > weekend in my 30 node cluster. I use ES as a DB so there were > constant writes while the node was down. Restarted the node and went > RED. > > [2013-03-10 15:28:22,209][WARN][indices.cluster] > [moloches-m13b] [stats][0] failed to start shard > org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: > [stats][0] shard allocated for local recovery (post api), should > exists, but doesn't Hmm, I wonder if there is a problem with the data store on that node? Perhaps just delete the datastore for that index on that node, and restart. clint -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Andy\_Wick](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andy_wick/32/44017_2.png) [@Andy\_Wick](https://discuss.elastic.co/u/Andy_Wick)\ **Post date:** [March 11, 2013, 3:30pm UTC](https://discuss.elastic.co/t/es-0-20-5-stuck-in-red-after-node-loss-or-how-do-i-configured-to-avoid-problems/11096/3 "2013-03-11T15:30:55Z")
Stopped the node, deleted the directory, restarted the node and then immediately two other nodes started having the same issue with the same index. Eventually I just gave up and deleted the index (I should say I deleted all the directories because -XDELETE on the index would just hang, I guess because it was in RED.) Should this work, or am I missing the point of replication? To me it seems like if one node is bad ES should just clean up and copy over a good version automatically. Thanks, Andy -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Clinton\_Gormley](https://avatars.discourse-cdn.com/v4/letter/c/50afbb/32.png) [@Clinton\_Gormley](https://discuss.elastic.co/u/Clinton_Gormley)\ **Post date:** [March 11, 2013, 3:57pm UTC](https://discuss.elastic.co/t/es-0-20-5-stuck-in-red-after-node-loss-or-how-do-i-configured-to-avoid-problems/11096/4 "2013-03-11T15:57:19Z")
On Mon, 2013-03-11 at 08:30 -0700, Andy Wick wrote: > Stopped the node, deleted the directory, restarted the node and then > immediately two other nodes started having the same issue with the > same index. Eventually I just gave up and deleted the index (I should > say I deleted all the directories because -XDELETE on the index would > just hang, I guess because it was in RED.) Should this work, or am I > missing the point of replication? To me it seems like if one node is > bad ES should just clean up and copy over a good version > automatically. Yes it should work. Please can you open an issue giving a full description, plus the full logs? ta clint -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 2:47am UTC](https://discuss.elastic.co/t/es-0-20-5-stuck-in-red-after-node-loss-or-how-do-i-configured-to-avoid-problems/11096/5 "2017-07-06T02:47:09Z")