# Did ES just nuke my index? **URL:** \ **Category:** Elasticsearch\ **Created:** [March 8, 2012, 12:27am UTC](https://discuss.elastic.co/t/did-es-just-nuke-my-index/6938 "2012-03-08T00:27:59Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![orenmazor](https://avatars.discourse-cdn.com/v4/letter/o/b782af/32.png) [@orenmazor](https://discuss.elastic.co/u/orenmazor)\ **Post date:** [March 8, 2012, 12:27am UTC](https://discuss.elastic.co/t/did-es-just-nuke-my-index/6938/1 "2012-03-08T00:27:59Z")
Hey all, I think my cluster just nuked my main index. I noticed queries were not coming back from my cluster. I happened to have bigdesk open, and noticed the cpu was pegged on both node machines with load just climbing. I killed the java process (which I've done quite a few times before… it hasn't caused failure in tests before), but upon restart my logs were full of this: [2012-03-07 18:32:25,735][WARN][cluster.action.shard] [Suprema] sending failed shard for [postmark\_staging][2], node[QpuhZarSRlWadRSaes4qAQ], [P], s[INITIALIZING], reason [Failed to start shard, message [IndexShardGatewayRecoveryException[[postmark\_staging][2] shard allocated for local recovery (post api), should exists, but doesn't]]] whats more, my cluster would not come up at all, so I couldn't admin it with the API. the frontend was showing the red status, but not actually giving me any option to act on it. I ended up moving the old node data aside into a backup and upon restart, ES came back up happy (of course without the index data…). I started my index rebuild while I debug the old data, since I suspect I might have to do this anyway… thoughts? can I recover this? my system setup: java: min/max ram is 16gb (10gb on the other node) SurvivorRatio is 6 (I found performance really improved once I tweaked this up) current index settings: ``` "index.number_of_replicas": "0", "index.number_of_shards": "10", "index.merge.policy.segments_per_tier": "20", "index.refresh_interval": "-1" ``` now, its also worth mentioning that we do a fairly high rate of deletes and indexing (all bulk, around 500 every 5-10 seconds). in my previous index settings, I also tweaked the ratio of merge deletes per segment (its 10 by default, I experimented upping it to 50 in some cases, and settled on 40 eventually). ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [March 8, 2012, 8:22pm UTC](https://discuss.elastic.co/t/did-es-just-nuke-my-index/6938/2 "2012-03-08T20:22:39Z")
Which version are you using? On Thursday, March 8, 2012 at 2:27 AM, Oren Mazor wrote: > Hey all, > > I think my cluster just nuked my main index. I noticed queries were not coming back from my cluster. I happened to have bigdesk open, and noticed the cpu was pegged on both node machines with load just climbing. > > I killed the java process (which I've done quite a few times before… it hasn't caused failure in tests before), but upon restart my logs were full of this: > > [2012-03-07 18:32:25,735][WARN][cluster.action.shard] [Suprema] sending failed shard for [postmark\_staging][2], node[QpuhZarSRlWadRSaes4qAQ], [P], s[INITIALIZING], reason [Failed to start shard, message [IndexShardGatewayRecoveryException[[postmark\_staging][2] shard allocated for local recovery (post api), should exists, but doesn't]]] > > whats more, my cluster would not come up at all, so I couldn't admin it with the API. the frontend was showing the red status, but not actually giving me any option to act on it. I ended up moving the old node data aside into a backup and upon restart, ES came back up happy (of course without the index data…). I started my index rebuild while I debug the old data, since I suspect I might have to do this anyway… > > thoughts? can I recover this? > > my system setup: > > java: > min/max ram is 16gb (10gb on the other node) > SurvivorRatio is 6 (I found performance really improved once I tweaked this up) > > current index settings: > "index.number\_of\_replicas": "0", "index.number\_of\_shards": "10", "index.merge.policy.segments\_per\_tier": "20", "index.refresh\_interval": "-1" > > now, its also worth mentioning that we do a fairly high rate of deletes and indexing (all bulk, around 500 every 5-10 seconds). in my previous index settings, I also tweaked the ratio of merge deletes per segment (its 10 by default, I experimented upping it to 50 in some cases, and settled on 40 eventually). ---
**Author:** ![orenmazor](https://avatars.discourse-cdn.com/v4/letter/o/b782af/32.png) [@orenmazor](https://discuss.elastic.co/u/orenmazor)\ **Post date:** [March 9, 2012, 12:31am UTC](https://discuss.elastic.co/t/did-es-just-nuke-my-index/6938/3 "2012-03-09T00:31:36Z")
Hey Shay, I'm running 0.18.6. After some googling, I saw there were issues with previous versions, but those issues were closed, so I assumed it's not related. On Thursday, March 8, 2012 3:22:39 PM UTC-5, kimchy wrote: > Which version are you using? > > On Thursday, March 8, 2012 at 2:27 AM, Oren Mazor wrote: > > Hey all, > > I think my cluster just nuked my main index. I noticed queries were not > coming back from my cluster. I happened to have bigdesk open, and noticed > the cpu was pegged on both node machines with load just climbing. > > I killed the java process (which I've done quite a few times before… it > hasn't caused failure in tests before), but upon restart my logs were full > of this: > > [2012-03-07 18:32:25,735][WARN][cluster.action.shard] [Suprema] > sending failed shard for [postmark\_staging][2], > node[QpuhZarSRlWadRSaes4qAQ], [P], s[INITIALIZING], reason [Failed to start > shard, message [IndexShardGatewayRecoveryException[[postmark\_staging][2] > shard allocated for local recovery (post api), should exists, but doesn't]]] > > whats more, my cluster would not come up at all, so I couldn't admin it > with the API. the frontend was showing the red status, but not actually > giving me any option to act on it. I ended up moving the old node data > aside into a backup and upon restart, ES came back up happy (of course > without the index data…). I started my index rebuild while I debug the old > data, since I suspect I might have to do this anyway… > > thoughts? can I recover this? > > my system setup: > > java: > min/max ram is 16gb (10gb on the other node) > SurvivorRatio is 6 (I found performance really improved once I tweaked > this up) > > current index settings: > > ``` > "index.number_of_replicas": "0", > "index.number_of_shards": "10", > "index.merge.policy.segments_per_tier": "20", > "index.refresh_interval": "-1" > > ``` > > now, its also worth mentioning that we do a fairly high rate of deletes and indexing (all bulk, around 500 every 5-10 seconds). in my previous index settings, I also tweaked the ratio of merge deletes per segment (its 10 by default, I experimented upping it to 50 in some cases, and settled on 40 eventually). ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [March 9, 2012, 7:02pm UTC](https://discuss.elastic.co/t/did-es-just-nuke-my-index/6938/4 "2012-03-09T19:02:11Z")
There was a bug fixed in 0.19 that might cause this in very rare cases (the cluster state one). On Friday, March 9, 2012 at 2:31 AM, Oren Mazor wrote: > Hey Shay, > > I'm running 0.18.6. After some googling, I saw there were issues with previous versions, but those issues were closed, so I assumed it's not related. > > On Thursday, March 8, 2012 3:22:39 PM UTC-5, kimchy wrote: > > > Which version are you using? > > > > On Thursday, March 8, 2012 at 2:27 AM, Oren Mazor wrote: > > > > > Hey all, > > > > > > I think my cluster just nuked my main index. I noticed queries were not coming back from my cluster. I happened to have bigdesk open, and noticed the cpu was pegged on both node machines with load just climbing. > > > > > > I killed the java process (which I've done quite a few times before… it hasn't caused failure in tests before), but upon restart my logs were full of this: > > > > > > [2012-03-07 18:32:25,735][WARN][cluster.action.shard] [Suprema] sending failed shard for [postmark\_staging][2], node[QpuhZarSRlWadRSaes4qAQ], [P], s[INITIALIZING], reason [Failed to start shard, message [IndexShardGatewayRecoveryException[[postmark\_staging][2] shard allocated for local recovery (post api), should exists, but doesn't]]] > > > > > > whats more, my cluster would not come up at all, so I couldn't admin it with the API. the frontend was showing the red status, but not actually giving me any option to act on it. I ended up moving the old node data aside into a backup and upon restart, ES came back up happy (of course without the index data…). I started my index rebuild while I debug the old data, since I suspect I might have to do this anyway… > > > > > > thoughts? can I recover this? > > > > > > my system setup: > > > > > > java: > > > min/max ram is 16gb (10gb on the other node) > > > SurvivorRatio is 6 (I found performance really improved once I tweaked this up) > > > > > > current index settings: > > > "index.number\_of\_replicas": "0", "index.number\_of\_shards": "10", "index.merge.policy.segments\_per\_tier": "20", "index.refresh\_interval": "-1" > > > > > > now, its also worth mentioning that we do a fairly high rate of deletes and indexing (all bulk, around 500 every 5-10 seconds). in my previous index settings, I also tweaked the ratio of merge deletes per segment (its 10 by default, I experimented upping it to 50 in some cases, and settled on 40 eventually). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 3:36am UTC](https://discuss.elastic.co/t/did-es-just-nuke-my-index/6938/5 "2017-07-06T03:36:32Z")