# Big issue **URL:** \ **Category:** Elasticsearch\ **Created:** [April 20, 2012, 3:58am UTC](https://discuss.elastic.co/t/big-issue/7412 "2012-04-20T03:58:04Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![jacque74](https://avatars.discourse-cdn.com/v4/letter/j/4af34b/32.png) [@jacque74](https://discuss.elastic.co/u/jacque74)\ **Post date:** [April 20, 2012, 3:58am UTC](https://discuss.elastic.co/t/big-issue/7412/1 "2012-04-20T03:58:04Z")
Hello, after some network problems, and what seemed like split brian , we ended up with this state: { "cluster\_name" : "elasticsearch0", "status" : "red", "timed\_out" : false, "number\_of\_nodes" : 11, "number\_of\_data\_nodes" : 11, "active\_primary\_shards" : 212, "active\_shards" : 212, "relocating\_shards" : 1, "initializing\_shards" : 0, "unassigned\_shards" : 38 } Note that we have 38 unassigned\_shares, and 0 initializing shards. We have all the data (from backup), but it looks like out of 11 servers running we are missing shards being assigned in their totality to the nodes. Again, I have all my index/ data saved, even those 38 shards are setting under /index in various nodes. However, without any errors in the error log, we simple can't exist the RED state. There is a way to repair metadata somehow? Thanks, -Jack ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [April 20, 2012, 8:38am UTC](https://discuss.elastic.co/t/big-issue/7412/2 "2012-04-20T08:38:08Z")
Are you saying you recovered the data from back and those shards don't initialize? Make sure to recover the \_state directory under the data location (nodes/0), especially if you are using 0.18. On Fri, Apr 20, 2012 at 6:58 AM, Jack Levin [magnito@gmail.com](mailto:magnito@gmail.com) wrote: > Hello, after some network problems, and what seemed like split brian , > we ended up with this state: > > { > "cluster\_name" : "elasticsearch0", > "status" : "red", > "timed\_out" : false, > "number\_of\_nodes" : 11, > "number\_of\_data\_nodes" : 11, > "active\_primary\_shards" : 212, > "active\_shards" : 212, > "relocating\_shards" : 1, > "initializing\_shards" : 0, > "unassigned\_shards" : 38 > } > > Note that we have 38 unassigned\_shares, and 0 initializing shards. We > have all the data (from backup), but it looks like out of 11 servers > running we are missing shards being assigned in their totality to the > nodes. Again, I have all my index/ data saved, even those 38 shards > are setting under /index in various nodes. However, without any > errors in the error log, we simple can't exist the RED state. There > is a way to repair metadata somehow? > > Thanks, > -Jack ---
**Author:** ![jacque74](https://avatars.discourse-cdn.com/v4/letter/j/4af34b/32.png) [@jacque74](https://discuss.elastic.co/u/jacque74)\ **Post date:** [April 20, 2012, 6:35pm UTC](https://discuss.elastic.co/t/big-issue/7412/3 "2012-04-20T18:35:37Z")
Shay, my worry is that \_state is corrupted, and is not indicating that a node has all the shards I have data for. Is there a way to edit \_state? I can move up a version (from .18) if needed to get it fixed. (longer story -- during the outage, backup script overwritten \_state possible with wrong values as it ran right when an outage happened). -Jack On Apr 20, 1:38 am, Shay Banon [kim...@gmail.com](mailto:kim...@gmail.com) wrote: > Are you saying you recovered the data from back and those shards don't > initialize? Make sure to recover the \_state directory under the data > location (nodes/0), especially if you are using 0.18. > > On Fri, Apr 20, 2012 at 6:58 AM, Jack Levin [magn...@gmail.com](mailto:magn...@gmail.com) wrote: > > > Hello, after some network problems, and what seemed like split brian , > > we ended up with this state: > > > { > > "cluster\_name" : "elasticsearch0", > > "status" : "red", > > "timed\_out" : false, > > "number\_of\_nodes" : 11, > > "number\_of\_data\_nodes" : 11, > > "active\_primary\_shards" : 212, > > "active\_shards" : 212, > > "relocating\_shards" : 1, > > "initializing\_shards" : 0, > > "unassigned\_shards" : 38 > > } > > > Note that we have 38 unassigned\_shares, and 0 initializing shards. We > > have all the data (from backup), but it looks like out of 11 servers > > running we are missing shards being assigned in their totality to the > > nodes. Again, I have all my index/ data saved, even those 38 shards > > are setting under /index in various nodes. However, without any > > errors in the error log, we simple can't exist the RED state. There > > is a way to repair metadata somehow? > > > Thanks, > > -Jack ---
**Author:** ![jacque74](https://avatars.discourse-cdn.com/v4/letter/j/4af34b/32.png) [@jacque74](https://discuss.elastic.co/u/jacque74)\ **Post date:** [April 20, 2012, 7:43pm UTC](https://discuss.elastic.co/t/big-issue/7412/4 "2012-04-20T19:43:23Z")
if we upgrade to 0.19.x perhaps it will solve the issue (https:// [Local Gateway: Move shard state to be stored under each shard, and not globally under \_state · Issue #1618 · elastic/elasticsearch · GitHub](http://github.com/elasticsearch/elasticsearch/issues/1618)).? -Jack On Apr 20, 11:35 am, Jack Levin [magn...@gmail.com](mailto:magn...@gmail.com) wrote: > Shay, my worry is that \_state is corrupted, and is not indicating that > a node has all the shards I have data for. Is there a way to edit > \_state? I can move up a version (from .18) if needed to get it fixed. > (longer story -- during the outage, backup script overwritten \_state > possible with wrong values as it ran right when an outage happened). > > -Jack > > On Apr 20, 1:38 am, Shay Banon [kim...@gmail.com](mailto:kim...@gmail.com) wrote: > > > Are you saying you recovered the data from back and those shards don't > > initialize? Make sure to recover the \_state directory under the data > > location (nodes/0), especially if you are using 0.18. > > > On Fri, Apr 20, 2012 at 6:58 AM, Jack Levin [magn...@gmail.com](mailto:magn...@gmail.com) wrote: > > > > > Hello, after some network problems, and what seemed like split brian , > > > we ended up with this state: > > > > { > > > "cluster\_name" : "elasticsearch0", > > > "status" : "red", > > > "timed\_out" : false, > > > "number\_of\_nodes" : 11, > > > "number\_of\_data\_nodes" : 11, > > > "active\_primary\_shards" : 212, > > > "active\_shards" : 212, > > > "relocating\_shards" : 1, > > > "initializing\_shards" : 0, > > > "unassigned\_shards" : 38 > > > } > > > > Note that we have 38 unassigned\_shares, and 0 initializing shards. We > > > have all the data (from backup), but it looks like out of 11 servers > > > running we are missing shards being assigned in their totality to the > > > nodes. Again, I have all my index/ data saved, even those 38 shards > > > are setting under /index in various nodes. However, without any > > > errors in the error log, we simple can't exist the RED state. There > > > is a way to repair metadata somehow? > > > > Thanks, > > > -Jack ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [April 21, 2012, 3:13pm UTC](https://discuss.elastic.co/t/big-issue/7412/5 "2012-04-21T15:13:45Z")
Hi, Yes, 0.19 local storage structure is considerably better compared to 0.18. The upgrade though will use the same problematic \_state doc that you have. What you can potentially do is upgrade to 0.19, and then, copy over the missing shards back some node, copy over the specific \_state dir from a working shard to the ones that are missing, edit the json file in the state to match the missing shard, and then see if things work... (in 0.19, each shard has its own allocation state). On Fri, Apr 20, 2012 at 10:43 PM, Jack Levin [magnito@gmail.com](mailto:magnito@gmail.com) wrote: > if we upgrade to 0.19.x perhaps it will solve the issue (https:// > [Local Gateway: Move shard state to be stored under each shard, and not globally under \_state · Issue #1618 · elastic/elasticsearch · GitHub](http://github.com/elasticsearch/elasticsearch/issues/1618)).? > > -Jack > > On Apr 20, 11:35 am, Jack Levin [magn...@gmail.com](mailto:magn...@gmail.com) wrote: > > > Shay, my worry is that \_state is corrupted, and is not indicating that > > a node has all the shards I have data for. Is there a way to edit > > \_state? I can move up a version (from .18) if needed to get it fixed. > > (longer story -- during the outage, backup script overwritten \_state > > possible with wrong values as it ran right when an outage happened). > > > > -Jack > > > > On Apr 20, 1:38 am, Shay Banon [kim...@gmail.com](mailto:kim...@gmail.com) wrote: > > > > > Are you saying you recovered the data from back and those shards don't > > > initialize? Make sure to recover the \_state directory under the data > > > location (nodes/0), especially if you are using 0.18. > > > > > On Fri, Apr 20, 2012 at 6:58 AM, Jack Levin [magn...@gmail.com](mailto:magn...@gmail.com) wrote: > > > > > > > Hello, after some network problems, and what seemed like split brian > > > > , > > > > we ended up with this state: > > > > > > { > > > > "cluster\_name" : "elasticsearch0", > > > > "status" : "red", > > > > "timed\_out" : false, > > > > "number\_of\_nodes" : 11, > > > > "number\_of\_data\_nodes" : 11, > > > > "active\_primary\_shards" : 212, > > > > "active\_shards" : 212, > > > > "relocating\_shards" : 1, > > > > "initializing\_shards" : 0, > > > > "unassigned\_shards" : 38 > > > > } > > > > > > Note that we have 38 unassigned\_shares, and 0 initializing shards. We > > > > have all the data (from backup), but it looks like out of 11 servers > > > > running we are missing shards being assigned in their totality to the > > > > nodes. Again, I have all my index/ data saved, even those 38 shards > > > > are setting under /index in various nodes. However, without any > > > > errors in the error log, we simple can't exist the RED state. There > > > > is a way to repair metadata somehow? > > > > > > Thanks, > > > > -Jack ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 3:31am UTC](https://discuss.elastic.co/t/big-issue/7412/6 "2017-07-06T03:31:28Z")
Closed on July 6, 2017, 3:31am UTC