# Recovering after shard failure **URL:** \ **Category:** Elasticsearch\ **Created:** [June 4, 2012, 1:20pm UTC](https://discuss.elastic.co/t/recovering-after-shard-failure/7965 "2012-06-04T13:20:11Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![Andy\_Wick](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andy_wick/32/44017_2.png) [@Andy\_Wick](https://discuss.elastic.co/u/Andy_Wick)\ **Post date:** [June 4, 2012, 1:20pm UTC](https://discuss.elastic.co/t/recovering-after-shard-failure/7965/1 "2012-06-04T13:20:11Z")
I had a shard go crazy, it created over 50 thousand files and elasticsearch was complaining about file limits. So on that machine I deleted everything for that index in the nodes/0/indices/sessions directory and restarted the instance, hoping that shards on the other machines wouldn't be lost. After restarting, the shard on that node stays in unassigned state and the only thing in the logs is [2012-06-03 19:52:43,553][DEBUG][gateway.local] [es-m01a] [sessions][12]: not allocating, number\_of\_allocated\_shards\_found [0], required\_number [1] I have 20 nodes and use total\_shards\_per\_node = 1 to force even distribution. settings: { index.refresh\_interval: 60 index.number\_of\_replicas: 0 index.number\_of\_shards: 20 index.routing.allocation.total\_shards\_per\_node: 1 index.cache.field.type: soft index.version.created: 190499 } Is the only solution to delete the entire index? ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [June 8, 2012, 12:25pm UTC](https://discuss.elastic.co/t/recovering-after-shard-failure/7965/2 "2012-06-08T12:25:24Z")
You don't have replicas for the shards, so if you delete the single copy of a shard, there isn't really another copy for it to recover from. Regarding the "shard go crazy" and create many files, do you see anything in the logs? On Mon, Jun 4, 2012 at 3:20 PM, Andy Wick [andywick@gmail.com](mailto:andywick@gmail.com) wrote: > I had a shard go crazy, it created over 50 thousand files and > elasticsearch was complaining about file limits. So on that machine I > deleted everything for that index in the nodes/0/indices/sessions directory > and restarted the instance, hoping that shards on the other machines > wouldn't be lost. After restarting, the shard on that node stays in > unassigned state and the only thing in the logs is > > [2012-06-03 19:52:43,553][DEBUG][gateway.local] [es-m01a] > [sessions][12]: not allocating, number\_of\_allocated\_shards\_found [0], > required\_number [1] > > I have 20 nodes and use total\_shards\_per\_node = 1 to force even > distribution. > > settings: { > index.refresh\_interval: 60 > index.number\_of\_replicas: 0 > index.number\_of\_shards: 20 > index.routing.allocation.total\_shards\_per\_node: 1 > index.cache.field.type: soft > index.version.created: 190499 > } > > Is the only solution to delete the entire index? ---
**Author:** ![Andy\_Wick](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andy_wick/32/44017_2.png) [@Andy\_Wick](https://discuss.elastic.co/u/Andy_Wick)\ **Post date:** [June 8, 2012, 1:05pm UTC](https://discuss.elastic.co/t/recovering-after-shard-failure/7965/3 "2012-06-08T13:05:49Z")
On Friday, June 8, 2012 8:25:24 AM UTC-4, kimchy wrote: > You don't have replicas for the shards, so if you delete the single copy > of a shard, there isn't really another copy for it to recover from. Sorry I wasn't clear, I understand the data is gone. My problem is that shard refused to go from unassigned to assigned. I had to delete the whole index. I'm fine with there being a "hole" in my index from all the data was lost, I was just hoping for a way to not loose everything. Is this possible, or without replication if you loose a shard you really loose the whole index? > Regarding the "shard go crazy" and create many files, do you see anything > in the logs? So it just happen again last night. A single index on a single node went to over 80k files, while the rest of the nodes have around 800 files for that same index. This time instead of removing the directory I increased the max number of open files to 128000 and restarted the cluster (restarting just the node didn't work.) Eventually it recovered the shard and now that node has around 1000 files. Nothing useful in the logs, the first error is too many open files. Since it seems like it might happen every 5 days is there a particular log I should turn on to debug, or just all of them? These are new machines, with three differences from previous setup: 4x the memory, running 2 nodes per machine instead of 1, and I'm using 2 disks now for data - path.data: ["/disk1/data", "/disk2/data"] ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [June 11, 2012, 2:50pm UTC](https://discuss.elastic.co/t/recovering-after-shard-failure/7965/4 "2012-06-11T14:50:47Z")
On Fri, Jun 8, 2012 at 3:05 PM, Andy Wick [andywick@gmail.com](mailto:andywick@gmail.com) wrote: > On Friday, June 8, 2012 8:25:24 AM UTC-4, kimchy wrote: > > > You don't have replicas for the shards, so if you delete the single copy > > of a shard, there isn't really another copy for it to recover from. > > Sorry I wasn't clear, I understand the data is gone. My problem is that > shard refused to go from unassigned to assigned. I had to delete the whole > index. I'm fine with there being a "hole" in my index from all the data > was lost, I was just hoping for a way to not loose everything. Is this > possible, or without replication if you loose a shard you really loose the > whole index? Yea, currently, there isn't an option to force the unassigned shards to be allocated and be empty. The assumption here is that you might be able at one point to bring back a a node with those shards allocation, so we wait for it. > > Regarding the "shard go crazy" and create many files, do you see anything > > in the logs? > > So it just happen again last night. A single index on a single node went > to over 80k files, while the rest of the nodes have around 800 files for > that same index. This time instead of removing the directory I increased > the max number of open files to 128000 and restarted the cluster > (restarting just the node didn't work.) Eventually it recovered the shard > and now that node has around 1000 files. > > Nothing useful in the logs, the first error is too many open files. Since > it seems like it might happen every 5 days is there a particular log I > should turn on to debug, or just all of them? > > These are new machines, with three differences from previous setup: 4x the > memory, running 2 nodes per machine instead of 1, and I'm using 2 disks now > for data - path.data: ["/disk1/data", "/disk2/data"] Can you gist the lsof -p [es\_process\_id] output when it happens again? Lets see what it has open. > ---
**Author:** ![Andy\_Wick](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andy_wick/32/44017_2.png) [@Andy\_Wick](https://discuss.elastic.co/u/Andy_Wick)\ **Post date:** [June 12, 2012, 8:42pm UTC](https://discuss.elastic.co/t/recovering-after-shard-failure/7965/5 "2012-06-12T20:42:50Z")
On Monday, June 11, 2012 10:50:47 AM UTC-4, kimchy wrote: > On Fri, Jun 8, 2012 at 3:05 PM, Andy Wick [andywick@gmail.com](mailto:andywick@gmail.com) wrote: > > > On Friday, June 8, 2012 8:25:24 AM UTC-4, kimchy wrote: > > > > > You don't have replicas for the shards, so if you delete the single copy > > > of a shard, there isn't really another copy for it to recover from. > > > > Sorry I wasn't clear, I understand the data is gone. My problem is that > > shard refused to go from unassigned to assigned. I had to delete the whole > > index. I'm fine with there being a "hole" in my index from all the data > > was lost, I was just hoping for a way to not loose everything. Is this > > possible, or without replication if you loose a shard you really loose the > > whole index? > > Yea, currently, there isn't an option to force the unassigned shards to be > allocated and be empty. The assumption here is that you might be able at > one point to bring back a a node with those shards allocation, so we wait > for it. > > > Can I put a feature request in? Or is there a minimum file set I can just backup and restore if this happens? > > > > Can you gist the lsof -p [es\_process\_id] output when it happens again? > Lets see what it has open. I caught it before it ran out of open files at about 40k files. Here are the first 2000 lines, the rest is more of the same. > All the new files are on /disk2, so seems like the issue might be that I'm using 2 data paths. I don't think it is related, because I have a fair number of these, but 3 minutes before the last timestamp in /disk1 I had a bulk exception because part of my data wasn't clean [2012-06-12 15:24:04,112][DEBUG][action.bulk] [moloches-m05a] [sessions][4] failed to bulk item (index) index {[REMOVED]]}]} org.elasticsearch.index.mapper.MapperParsingException: Failed to parse [ho] at org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(AbstractFieldMapper.java:325) at org.elasticsearch.index.mapper.object.ObjectMapper.serializeValue(ObjectMapper.java:585) at org.elasticsearch.index.mapper.object.ObjectMapper.serializeArray(ObjectMapper.java:573) at org.elasticsearch.index.mapper.object.ObjectMapper.parse(ObjectMapper.java:441) at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:493) at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:437) at org.elasticsearch.index.shard.service.InternalIndexShard.prepareIndex(InternalIndexShard.java:311) at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:156) at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.performOnPrimary(TransportShardReplicationOperationAction.java:532) at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1.run(TransportShardReplicationOperationAction.java:430) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Caused by: org.elasticsearch.common.jackson.JsonParseException: Unrecognized character escape (CTRL-CHAR, code 8) at [Source: [B@39a06e34; line: 1, column: 228] at org.elasticsearch.common.jackson.JsonParser.\_constructError(JsonParser.java:1433) at org.elasticsearch.common.jackson.impl.JsonParserMinimalBase.\_reportError(JsonParserMinimalBase.java:521) at org.elasticsearch.common.jackson.impl.JsonParserMinimalBase.\_handleUnrecognizedCharacterEscape(JsonParserMinimalBase.java:496) at org.elasticsearch.common.jackson.impl.Utf8StreamParser.\_decodeEscaped(Utf8StreamParser.java:2540) at org.elasticsearch.common.jackson.impl.Utf8StreamParser.\_finishString2(Utf8StreamParser.java:1949) at org.elasticsearch.common.jackson.impl.Utf8StreamParser.\_finishString(Utf8StreamParser.java:1905) at org.elasticsearch.common.jackson.impl.Utf8StreamParser.getText(Utf8StreamParser.java:276) at org.elasticsearch.common.xcontent.json.JsonXContentParser.text(JsonXContentParser.java:83) at org.elasticsearch.common.xcontent.support.AbstractXContentParser.textOrNull(AbstractXContentParser.java:106) at org.elasticsearch.index.mapper.core.StringFieldMapper.parseCreateField(StringFieldMapper.java:242) at org.elasticsearch.index.mapper.core.StringFieldMapper.parseCreateField(StringFieldMapper.java:43) at org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(AbstractFieldMapper.java:312) ... 12 more ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [June 14, 2012, 8:41pm UTC](https://discuss.elastic.co/t/recovering-after-shard-failure/7965/6 "2012-06-14T20:41:16Z")
On Tue, Jun 12, 2012 at 10:42 PM, Andy Wick [andywick@gmail.com](mailto:andywick@gmail.com) wrote: > On Monday, June 11, 2012 10:50:47 AM UTC-4, kimchy wrote: > > > On Fri, Jun 8, 2012 at 3:05 PM, Andy Wick [andywick@gmail.com](mailto:andywick@gmail.com) wrote: > > > > > On Friday, June 8, 2012 8:25:24 AM UTC-4, kimchy wrote: > > > > > > > You don't have replicas for the shards, so if you delete the single > > > > copy of a shard, there isn't really another copy for it to recover from. > > > > > > Sorry I wasn't clear, I understand the data is gone. My problem is that > > > shard refused to go from unassigned to assigned. I had to delete the whole > > > index. I'm fine with there being a "hole" in my index from all the data > > > was lost, I was just hoping for a way to not loose everything. Is this > > > possible, or without replication if you loose a shard you really loose the > > > whole index? > > > > Yea, currently, there isn't an option to force the unassigned shards to > > be allocated and be empty. The assumption here is that you might be able at > > one point to bring back a a node with those shards allocation, so we wait > > for it. > > > > > > > Can I put a feature request in? Or is there a minimum file set I can just > backup and restore if this happens? Yea, you can. A quick hack can be to create a similar empty index locally and copy over the relevant missing shard to a node data location, and create a dummy index (just to get relocation happening again). > > > > > > > Can you gist the lsof -p [es\_process\_id] output when it happens again? > > Lets see what it has open. > > I caught it before it ran out of open files at about 40k files. Here are > the first 2000 lines, the rest is more of the same. > > [ElasticSearch many files · GitHub](https://gist.github.com/2919936) > > All the new files are on /disk2, so seems like the issue might be that I'm > using 2 data paths. - Do you have any special configuration on the sessions index? Custom merge policy settings perhaps? - Which ES version are you using? - Can you try with one data path and see if the problem goes away? (I don't think that thats the problem). > I don't think it is related, because I have a fair number of these, but 3 > minutes before the last timestamp in /disk1 I had a bulk exception because > part of my data wasn't clean > > [2012-06-12 15:24:04,112][DEBUG][action.bulk] > [moloches-m05a] [sessions][4] failed to bulk item (index) index > {[REMOVED]]}]} > org.elasticsearch.index.mapper.MapperParsingException: Failed to parse [ho] > at > org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(AbstractFieldMapper.java:325) > at > org.elasticsearch.index.mapper.object.ObjectMapper.serializeValue(ObjectMapper.java:585) > at > org.elasticsearch.index.mapper.object.ObjectMapper.serializeArray(ObjectMapper.java:573) > at > org.elasticsearch.index.mapper.object.ObjectMapper.parse(ObjectMapper.java:441) > at > org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:493) > at > org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:437) > at > org.elasticsearch.index.shard.service.InternalIndexShard.prepareIndex(InternalIndexShard.java:311) > at > org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:156) > at > org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.performOnPrimary(TransportShardReplicationOperationAction.java:532) > at > org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1.run(TransportShardReplicationOperationAction.java:430) > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown > Source) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown > Source) > at java.lang.Thread.run(Unknown Source) > Caused by: org.elasticsearch.common.jackson.JsonParseException: > Unrecognized character escape (CTRL-CHAR, code 8) > at [Source: [B@39a06e34; line: 1, column: 228] > at > org.elasticsearch.common.jackson.JsonParser.\_constructError(JsonParser.java:1433) > at > org.elasticsearch.common.jackson.impl.JsonParserMinimalBase.\_reportError(JsonParserMinimalBase.java:521) > at > org.elasticsearch.common.jackson.impl.JsonParserMinimalBase.\_handleUnrecognizedCharacterEscape(JsonParserMinimalBase.java:496) > at > org.elasticsearch.common.jackson.impl.Utf8StreamParser.\_decodeEscaped(Utf8StreamParser.java:2540) > at > org.elasticsearch.common.jackson.impl.Utf8StreamParser.\_finishString2(Utf8StreamParser.java:1949) > at > org.elasticsearch.common.jackson.impl.Utf8StreamParser.\_finishString(Utf8StreamParser.java:1905) > at > org.elasticsearch.common.jackson.impl.Utf8StreamParser.getText(Utf8StreamParser.java:276) > at > org.elasticsearch.common.xcontent.json.JsonXContentParser.text(JsonXContentParser.java:83) > at > org.elasticsearch.common.xcontent.support.AbstractXContentParser.textOrNull(AbstractXContentParser.java:106) > at > org.elasticsearch.index.mapper.core.StringFieldMapper.parseCreateField(StringFieldMapper.java:242) > at > org.elasticsearch.index.mapper.core.StringFieldMapper.parseCreateField(StringFieldMapper.java:43) > at > org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(AbstractFieldMapper.java:312) > ... 12 more Its simply a parser failure. But this message is strange failed to bulk item (index) index {[REMOVED]]}]}? Did you remove thigns there? ---
**Author:** ![Andy\_Wick](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andy_wick/32/44017_2.png) [@Andy\_Wick](https://discuss.elastic.co/u/Andy_Wick)\ **Post date:** [June 15, 2012, 12:12am UTC](https://discuss.elastic.co/t/recovering-after-shard-failure/7965/7 "2012-06-15T00:12:58Z")
> . > > > [ElasticSearch many files · GitHub](https://gist.github.com/2919936) > > > > All the new files are on /disk2, so seems like the issue might be that > > I'm using 2 data paths. > > - Do you have any special configuration on the sessions index? Custom > merge policy settings perhaps? I don't think so "sessions" : { "settings" : { "index.analysis.filter.url\_stop.stopwords.1" : "https", "index.analysis.filter.url\_stop.stopwords.0" : "http", "index.refresh\_interval" : "60", "index.version.created" : "190499", "index.number\_of\_shards" : "20", "index.analysis.filter.url\_stop.type" : "stop", "index.analysis.analyzer.url\_analyzer.filter.1" : "url\_stop", "index.analysis.analyzer.url\_analyzer.filter.0" : "stop", "index.analysis.analyzer.url\_analyzer.type" : "custom", "index.number\_of\_replicas" : "0", "index.analysis.analyzer.url\_analyzer.tokenizer" : "lowercase", "index.routing.allocation.total\_shards\_per\_node" : "1", "index.cache.field.type" : "soft" } } > - Which ES version are you using? 0.19.4 > - Can you try with one data path and see if the problem goes away? (I > don't think that thats the problem). Yes, I'll try that. > > I don't think it is related, because I have a fair number of these, but 3 > > minutes before the last timestamp in /disk1 I had a bulk exception because > > part of my data wasn't clean > > > > Its simply a parser failure. But this message is strange failed to bulk > > item (index) index {[REMOVED]]}]}? Did you remove thigns there? Yes, I can't share the actual data, sorry I wasn't more clear. Thanks, Andy ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 3:24am UTC](https://discuss.elastic.co/t/recovering-after-shard-failure/7965/8 "2017-07-06T03:24:01Z")