# ES failed to recover after crash **URL:** \ **Category:** Elasticsearch\ **Created:** [June 22, 2012, 1:36pm UTC](https://discuss.elastic.co/t/es-failed-to-recover-after-crash/8195 "2012-06-22T13:36:16Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![nelalx](https://avatars.discourse-cdn.com/v4/letter/n/7c8e57/32.png) [@nelalx](https://discuss.elastic.co/u/nelalx)\ **Post date:** [June 22, 2012, 1:36pm UTC](https://discuss.elastic.co/t/es-failed-to-recover-after-crash/8195/1 "2012-06-22T13:36:16Z")
Hi We are facing an isssue with ES. Our ES instance crashed due to low disk space. After that it triggers out exceptions which are described below.. at org.elasticsearch.index.shard.service.InternalIndexShard.performRecoveryOperation(InternalIndexShard.java:624) at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:196) ... 4 more Caused by: org.elasticsearch.common.jackson.JsonParseException: Illegal unquoted character ((CTRL-CHAR, code 0)): has to be escaped using backslash to be included in string value at [Source: [B@bfc25c; line: 1, column: 4062] at org.elasticsearch.common.jackson.JsonParser.\_constructError(JsonParser.java:1432) at org.elasticsearch.common.jackson.impl.JsonParserMinimalBase.\_reportError(JsonParserMinimalBase.java:521) at org.elasticsearch.common.jackson.impl.JsonParserMinimalBase.\_throwUnquotedSpace(JsonParserMinimalBase.java:482) at org.elasticsearch.common.jackson.impl.Utf8StreamParser.\_finishString2(Utf8StreamParser.java:1975) at org.elasticsearch.common.jackson.impl.Utf8StreamParser.\_finishString(Utf8StreamParser.java:1905) at org.elasticsearch.common.jackson.impl.Utf8StreamParser.getText(Utf8StreamParser.java:276) at org.elasticsearch.common.xcontent.json.JsonXContentParser.text(JsonXContentParser.java:83) at org.elasticsearch.common.xcontent.support.AbstractXContentParser.textOrNull(AbstractXContentParser.java:106) at org.elasticsearch.index.mapper.core.StringFieldMapper.parseCreateField(StringFieldMapper.java:207) at org.elasticsearch.index.mapper.core.StringFieldMapper.parseCreateField(StringFieldMapper.java:42) at org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(AbstractFieldMapper.java:307) ... 11 more [2012-06-22 17:36:11,075][WARN][cluster.action.shard] [Cat-Man] sending failed shard for [algotree][0], node[Rl9eKHxLRe6PDD9am-L9og], [P], s[INITIALIZING], reason [Failed to start shard, message [IndexShardGatewayRecoveryException[[algotree][0] failed to recover shard]; nested: MapperParsingException[Failed to parse [Link]]; nested: JsonParseException[Illegal unquoted character ((CTRL-CHAR, code 0)): has to be escaped using backslash to be included in string value at [Source: [B@bfc25c; line: 1, column: 4062]]; ]] [2012-06-22 17:36:11,075][WARN][cluster.action.shard] [Cat-Man] received shard failed for [algotree][0], node[Rl9eKHxLRe6PDD9am-L9og], [P], s[INITIALIZING], reason [Failed to start shard, message [IndexShardGatewayRecoveryException[[algotree][0] failed to recover shard]; nested: MapperParsingException[Failed to parse [Link]]; nested: JsonParseException[Illegal unquoted character ((CTRL-CHAR, code 0)): has to be escaped using backslash to be included in string value at [Source: [B@bfc25c; line: 1, column: 4062]]; ]] [2012-06-22 17:36:13,107][WARN][indices.cluster] [Cat-Man] [algotree][1] failed to start shard org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [algotree][1] failed to recover shard at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:201) at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:177) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Caused by: org.elasticsearch.index.mapper.MapperParsingException: Failed to parse [Categories.Types.Facets.MoneyEvent.entity] at org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(AbstractFieldMapper.java:320) at org.elasticsearch.index.mapper.object.ObjectMapper.serializeValue(ObjectMapper.java:580) at org.elasticsearch.index.mapper.object.ObjectMapper.parse(ObjectMapper.java:444) at org.elasticsearch.index.mapper.object.ObjectMapper.serializeObject(ObjectMapper.java:492) at org.elasticsearch.index.mapper.object.ObjectMapper.serializeArray(ObjectMapper.java:560) at org.elasticsearch.index.mapper.object.ObjectMapper.parse(ObjectMapper.java:436) at org.elasticsearch.index.mapper.object.ObjectMapper.serializeObject(ObjectMapper.java:492) at org.elasticsearch.index.mapper.object.ObjectMapper.serializeArray(ObjectMapper.java:560) at org.elasticsearch.index.mapper.object.ObjectMapper.parse(ObjectMapper.java:436) at org.elasticsearch.index.mapper.object.ObjectMapper.serializeObject(ObjectMapper.java:492) at org.elasticsearch.index.mapper.object.ObjectMapper.parse(ObjectMapper.java:434) at org.elasticsearch.index.mapper.object.ObjectMapper.serializeObject(ObjectMapper.java:492) at org.elasticsearch.index.mapper.object.ObjectMapper.parse(ObjectMapper.java:434) at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:473) at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:417) at org.elasticsearch.index.shard.service.InternalIndexShard.prepareIndex(InternalIndexShard.java:311) at org.elasticsearch.index.shard.service.InternalIndexShard.performRecoveryOperation(InternalIndexShard.java:624) at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:196) ... 4 more Caused by: org.elasticsearch.common.jackson.JsonParseException: Illegal unquoted character ((CTRL-CHAR, code 0)): has to be escaped using backslash to be included in string value at [Source: [B@eb203b; line: 1, column: 937] at org.elasticsearch.common.jackson.JsonParser.\_constructError(JsonParser.java:1432) at org.elasticsearch.common.jackson.impl.JsonParserMinimalBase.\_reportError(JsonParserMinimalBase.java:521) at org.elasticsearch.common.jackson.impl.JsonParserMinimalBase.\_throwUnquotedSpace(JsonParserMinimalBase.java:482) at org.elasticsearch.common.jackson.impl.Utf8StreamParser.\_finishString2(Utf8StreamParser.java:1975) at org.elasticsearch.common.jackson.impl.Utf8StreamParser.\_finishString(Utf8StreamParser.java:1905) at org.elasticsearch.common.jackson.impl.Utf8StreamParser.getText(Utf8StreamParser.java:276) at org.elasticsearch.common.xcontent.json.JsonXContentParser.text(JsonXContentParser.java:83) at org.elasticsearch.common.xcontent.support.AbstractXContentParser.textOrNull(AbstractXContentParser.java:106) at org.elasticsearch.index.mapper.core.StringFieldMapper.parseCreateField(StringFieldMapper.java:207) at org.elasticsearch.index.mapper.core.StringFieldMapper.parseCreateField(StringFieldMapper.java:42) at org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(AbstractFieldMapper.java:307) ... 21 more [2012-06-22 17:36:13,135][WARN][cluster.action.shard] [Cat-Man] sending failed shard for [algotree][1], node[Rl9eKHxLRe6PDD9am-L9og], [P], s[INITIALIZING], reason [Failed to start shard, message [IndexShardGatewayRecoveryException[[algotree][1] failed to recover shard]; nested: MapperParsingException[Failed to parse [Categories.Types.Facets.MoneyEvent.entity]]; nested: JsonParseException[Illegal unquoted character ((CTRL-CHAR, code 0)): has to be escaped using backslash to be included in string value at [Source: [B@eb203b; line: 1, column: 937]]; ]] [2012-06-22 17:36:13,135][WARN][cluster.action.shard] [Cat-Man] received shard failed for [algotree][1], node[Rl9eKHxLRe6PDD9am-L9og], [P], s[INITIALIZING], reason [Failed to start shard, message [IndexShardGatewayRecoveryException[[algotree][1] failed to recover shard]; nested: MapperParsingException[Failed to parse [Categories.Types.Facets.MoneyEvent.entity]]; nested: JsonParseException[Illegal unquoted character ((CTRL-CHAR, code 0)): has to be escaped using backslash to be included in string value at [Source: [B@eb203b; line: 1, column: 937]]; ]] [2012-06-22 17:36:14,895][WARN][indices.cluster] [Cat-Man] [algotree][0] failed to start shard org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [algotree][0] failed to recover shard at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:201) at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:177) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Caused by: org.elasticsearch.index.mapper.MapperParsingException: Failed to parse [Link] at org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(AbstractFieldMapper.java:320) at org.elasticsearch.index.mapper.object.ObjectMapper.serializeValue(ObjectMapper.java:580) at org.elasticsearch.index.mapper.object.ObjectMapper.parse(ObjectMapper.java:444) at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:473) at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:417) at org.elasticsearch.index.shard.service.InternalIndexShard.prepareIndex(InternalIndexShard.java:311) at org.elasticsearch.index.shard.service.InternalIndexShard.performRecoveryOperation(InternalIndexShard.java:624) at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:196) ... 4 more Caused by: org.elasticsearch.common.jackson.JsonParseException: Illegal unquoted character ((CTRL-CHAR, code 0)): has to be escaped using backslash to be included in string value at [Source: [B@11d565f; line: 1, column: 4062] at org.elasticsearch.common.jackson.JsonParser.\_constructError(JsonParser.java:1432) at org.elasticsearch.common.jackson.impl.JsonParserMinimalBase.\_reportError(JsonParserMinimalBase.java:521) at org.elasticsearch.common.jackson.impl.JsonParserMinimalBase.\_throwUnquotedSpace(JsonParserMinimalBase.java:482) at org.elasticsearch.common.jackson.impl.Utf8StreamParser.\_finishString2(Utf8StreamParser.java:1975) at org.elasticsearch.common.jackson.impl.Utf8StreamParser.\_finishString(Utf8StreamParser.java:1905) at org.elasticsearch.common.jackson.impl.Utf8StreamParser.getText(Utf8StreamParser.java:276) at org.elasticsearch.common.xcontent.json.JsonXContentParser.text(JsonXContentParser.java:83) at org.elasticsearch.common.xcontent.support.AbstractXContentParser.textOrNull(AbstractXContentParser.java:106) at org.elasticsearch.index.mapper.core.StringFieldMapper.parseCreateField(StringFieldMapper.java:207) at org.elasticsearch.index.mapper.core.StringFieldMapper.parseCreateField(StringFieldMapper.java:42) at org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(AbstractFieldMapper.java:307) ... 11 more [2012-06-22 17:36:14,922][WARN][cluster.action.shard] [Cat-Man] sending failed shard for [algotree][0], node[Rl9eKHxLRe6PDD9am-L9og], [P], s[INITIALIZING], reason [Failed to start shard, message [IndexShardGatewayRecoveryException[[algotree][0] failed to recover shard]; nested: MapperParsingException[Failed to parse [Link]]; nested: JsonParseException[Illegal unquoted character ((CTRL-CHAR, code 0)): has to be escaped using backslash to be included in string value at [Source: [B@11d565f; line: 1, column: 4062]]; ]] [2012-06-22 17:36:14,922][WARN][cluster.action.shard] [Cat-Man] received shard failed for [algotree][0], node[Rl9eKHxLRe6PDD9am-L9og], [P], s[INITIALIZING], reason [Failed to start shard, message [IndexShardGatewayRecoveryException[[algotree][0] failed to recover shard]; nested: MapperParsingException[Failed to parse [Link]]; nested: JsonParseException[Illegal unquoted character ((CTRL-CHAR, code 0)): has to be escaped using backslash to be included in string value at [Source: [B@11d565f; line: 1, column: 4062]]; ]] [2012-06-22 17:36:17,165][WARN][indices.cluster] [Cat-Man] [algotree][1] failed to start shard org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [algotree][1] failed to recover shard at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:201) at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:177) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Caused by: org.elasticsearch.index.mapper.MapperParsingException: Failed to parse [Categories.Types.Facets.MoneyEvent.entity] at org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(AbstractFieldMapper.java:320) at org.elasticsearch.index.mapper.object.ObjectMapper.serializeValue(ObjectMapper.java:580) at org.elasticsearch.index.mapper.object.ObjectMapper.parse(ObjectMapper.java:444) at org.elasticsearch.index.mapper.object.ObjectMapper.serializeObject(ObjectMapper.java:492) at org.elasticsearch.index.mapper.object.ObjectMapper.serializeArray(ObjectMapper.java:560) at org.elasticsearch.index.mapper.object.ObjectMapper.parse(ObjectMapper.java:436) at org.elasticsearch.index.mapper.object.ObjectMapper.serializeObject(ObjectMapper.java:492) at org.elasticsearch.index.mapper.object.ObjectMapper.serializeArray(ObjectMapper.java:560) at org.elasticsearch.index.mapper.object.ObjectMapper.parse(ObjectMapper.java:436) at org.elasticsearch.index.mapper.object.ObjectMapper.serializeObject(ObjectMapper.java:492) at org.elasticsearch.index.mapper.object.ObjectMapper.parse(ObjectMapper.java:434) at org.elasticsearch.index.mapper.object.ObjectMapper.serializeObject(ObjectMapper.java:492) at org.elasticsearch.index.mapper.object.ObjectMapper.parse(ObjectMapper.java:434) at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:473) at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:417) at org.elasticsearch.index.shard.service.InternalIndexShard.prepareIndex(InternalIndexShard.java:311) at org.elasticsearch.index.shard.service.InternalIndexShard.performRecoveryOperation(InternalIndexShard.java:624) at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:196) ... 4 more . What could have posssibly happened? How can we resolve this issue.. we are using Ubuntu..is there a way that i can get the data back.. Thanks in advance...... nel ---
**Author:** ![Pierrick](https://avatars.discourse-cdn.com/v4/letter/p/58f4c7/32.png) [@Pierrick](https://discuss.elastic.co/u/Pierrick)\ **Post date:** [June 28, 2012, 7:57am UTC](https://discuss.elastic.co/t/es-failed-to-recover-after-crash/8195/2 "2012-06-28T07:57:46Z")
Hi, Same problem for me. Anyone has a solution ? On Friday, June 22, 2012 3:36:16 PM UTC+2, nel wrote: > Hi > > We are facing an isssue with ES. Our ES instance crashed due to low disk > space. After that it triggers out exceptions which are described below.. > > at > org.elasticsearch.index.shard.service.InternalIndexShard.performRecoveryOperation(InternalIndexShard.java:624) > at > org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:196) > ... 4 more > Caused by: org.elasticsearch.common.jackson.JsonParseException: Illegal > unquoted character ((CTRL-CHAR, code 0)): has to be escaped using backslash > to be included in string value > at [Source: [B@bfc25c; line: 1, column: 4062] > at > org.elasticsearch.common.jackson.JsonParser.\_constructError(JsonParser.java:1432) > at > org.elasticsearch.common.jackson.impl.JsonParserMinimalBase.\_reportError(JsonParserMinimalBase.java:521) > at > org.elasticsearch.common.jackson.impl.JsonParserMinimalBase.\_throwUnquotedSpace(JsonParserMinimalBase.java:482) > at > org.elasticsearch.common.jackson.impl.Utf8StreamParser.\_finishString2(Utf8StreamParser.java:1975) > at > org.elasticsearch.common.jackson.impl.Utf8StreamParser.\_finishString(Utf8StreamParser.java:1905) > at > org.elasticsearch.common.jackson.impl.Utf8StreamParser.getText(Utf8StreamParser.java:276) > at > org.elasticsearch.common.xcontent.json.JsonXContentParser.text(JsonXContentParser.java:83) > at > org.elasticsearch.common.xcontent.support.AbstractXContentParser.textOrNull(AbstractXContentParser.java:106) > at > org.elasticsearch.index.mapper.core.StringFieldMapper.parseCreateField(StringFieldMapper.java:207) > at > org.elasticsearch.index.mapper.core.StringFieldMapper.parseCreateField(StringFieldMapper.java:42) > at > org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(AbstractFieldMapper.java:307) > ... 11 more > [2012-06-22 17:36:11,075][WARN][cluster.action.shard] [Cat-Man] > sending failed shard for [algotree][0], node[Rl9eKHxLRe6PDD9am-L9og], [P], > s[INITIALIZING], reason [Failed to start shard, message > [IndexShardGatewayRecoveryException[[algotree][0] failed to recover shard]; > nested: MapperParsingException[Failed to parse [Link]]; nested: > JsonParseException[Illegal unquoted character ((CTRL-CHAR, code 0)): has to > be escaped using backslash to be included in string value > at [Source: [B@bfc25c; line: 1, column: 4062]]; ]] > [2012-06-22 17:36:11,075][WARN][cluster.action.shard] [Cat-Man] > received shard failed for [algotree][0], node[Rl9eKHxLRe6PDD9am-L9og], [P], > s[INITIALIZING], reason [Failed to start shard, message > [IndexShardGatewayRecoveryException[[algotree][0] failed to recover shard]; > nested: MapperParsingException[Failed to parse [Link]]; nested: > JsonParseException[Illegal unquoted character ((CTRL-CHAR, code 0)): has to > be escaped using backslash to be included in string value > at [Source: [B@bfc25c; line: 1, column: 4062]]; ]] > [2012-06-22 17:36:13,107][WARN][indices.cluster] [Cat-Man] > [algotree][1] failed to start shard > org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: > [algotree][1] failed to recover shard > at > org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:201) > at > org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:177) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > Caused by: org.elasticsearch.index.mapper.MapperParsingException: Failed > to parse [Categories.Types.Facets.MoneyEvent.entity] > at > org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(AbstractFieldMapper.java:320) > at > org.elasticsearch.index.mapper.object.ObjectMapper.serializeValue(ObjectMapper.java:580) > at > org.elasticsearch.index.mapper.object.ObjectMapper.parse(ObjectMapper.java:444) > at > org.elasticsearch.index.mapper.object.ObjectMapper.serializeObject(ObjectMapper.java:492) > at > org.elasticsearch.index.mapper.object.ObjectMapper.serializeArray(ObjectMapper.java:560) > at > org.elasticsearch.index.mapper.object.ObjectMapper.parse(ObjectMapper.java:436) > at > org.elasticsearch.index.mapper.object.ObjectMapper.serializeObject(ObjectMapper.java:492) > at > org.elasticsearch.index.mapper.object.ObjectMapper.serializeArray(ObjectMapper.java:560) > at > org.elasticsearch.index.mapper.object.ObjectMapper.parse(ObjectMapper.java:436) > at > org.elasticsearch.index.mapper.object.ObjectMapper.serializeObject(ObjectMapper.java:492) > at > org.elasticsearch.index.mapper.object.ObjectMapper.parse(ObjectMapper.java:434) > at > org.elasticsearch.index.mapper.object.ObjectMapper.serializeObject(ObjectMapper.java:492) > at > org.elasticsearch.index.mapper.object.ObjectMapper.parse(ObjectMapper.java:434) > at > org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:473) > at > org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:417) > at > org.elasticsearch.index.shard.service.InternalIndexShard.prepareIndex(InternalIndexShard.java:311) > at > org.elasticsearch.index.shard.service.InternalIndexShard.performRecoveryOperation(InternalIndexShard.java:624) > at > org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:196) > ... 4 more > Caused by: org.elasticsearch.common.jackson.JsonParseException: Illegal > unquoted character ((CTRL-CHAR, code 0)): has to be escaped using backslash > to be included in string value > at [Source: [B@eb203b; line: 1, column: 937] > at > org.elasticsearch.common.jackson.JsonParser.\_constructError(JsonParser.java:1432) > at > org.elasticsearch.common.jackson.impl.JsonParserMinimalBase.\_reportError(JsonParserMinimalBase.java:521) > at > org.elasticsearch.common.jackson.impl.JsonParserMinimalBase.\_throwUnquotedSpace(JsonParserMinimalBase.java:482) > at > org.elasticsearch.common.jackson.impl.Utf8StreamParser.\_finishString2(Utf8StreamParser.java:1975) > at > org.elasticsearch.common.jackson.impl.Utf8StreamParser.\_finishString(Utf8StreamParser.java:1905) > at > org.elasticsearch.common.jackson.impl.Utf8StreamParser.getText(Utf8StreamParser.java:276) > at > org.elasticsearch.common.xcontent.json.JsonXContentParser.text(JsonXContentParser.java:83) > at > org.elasticsearch.common.xcontent.support.AbstractXContentParser.textOrNull(AbstractXContentParser.java:106) > at > org.elasticsearch.index.mapper.core.StringFieldMapper.parseCreateField(StringFieldMapper.java:207) > at > org.elasticsearch.index.mapper.core.StringFieldMapper.parseCreateField(StringFieldMapper.java:42) > at > org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(AbstractFieldMapper.java:307) > ... 21 more > [2012-06-22 17:36:13,135][WARN][cluster.action.shard] [Cat-Man] > sending failed shard for [algotree][1], node[Rl9eKHxLRe6PDD9am-L9og], [P], > s[INITIALIZING], reason [Failed to start shard, message > [IndexShardGatewayRecoveryException[[algotree][1] failed to recover shard]; > nested: MapperParsingException[Failed to parse > [Categories.Types.Facets.MoneyEvent.entity]]; nested: > JsonParseException[Illegal unquoted character ((CTRL-CHAR, code 0)): has to > be escaped using backslash to be included in string value > at [Source: [B@eb203b; line: 1, column: 937]]; ]] > [2012-06-22 17:36:13,135][WARN][cluster.action.shard] [Cat-Man] > received shard failed for [algotree][1], node[Rl9eKHxLRe6PDD9am-L9og], [P], > s[INITIALIZING], reason [Failed to start shard, message > [IndexShardGatewayRecoveryException[[algotree][1] failed to recover shard]; > nested: MapperParsingException[Failed to parse > [Categories.Types.Facets.MoneyEvent.entity]]; nested: > JsonParseException[Illegal unquoted character ((CTRL-CHAR, code 0)): has to > be escaped using backslash to be included in string value > at [Source: [B@eb203b; line: 1, column: 937]]; ]] > [2012-06-22 17:36:14,895][WARN][indices.cluster] [Cat-Man] > [algotree][0] failed to start shard > org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: > [algotree][0] failed to recover shard > at > org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:201) > at > org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:177) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > Caused by: org.elasticsearch.index.mapper.MapperParsingException: Failed > to parse [Link] > at > org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(AbstractFieldMapper.java:320) > at > org.elasticsearch.index.mapper.object.ObjectMapper.serializeValue(ObjectMapper.java:580) > at > org.elasticsearch.index.mapper.object.ObjectMapper.parse(ObjectMapper.java:444) > at > org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:473) > at > org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:417) > at > org.elasticsearch.index.shard.service.InternalIndexShard.prepareIndex(InternalIndexShard.java:311) > at > org.elasticsearch.index.shard.service.InternalIndexShard.performRecoveryOperation(InternalIndexShard.java:624) > at > org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:196) > ... 4 more > Caused by: org.elasticsearch.common.jackson.JsonParseException: Illegal > unquoted character ((CTRL-CHAR, code 0)): has to be escaped using backslash > to be included in string value > at [Source: [B@11d565f; line: 1, column: 4062] > at > org.elasticsearch.common.jackson.JsonParser.\_constructError(JsonParser.java:1432) > at > org.elasticsearch.common.jackson.impl.JsonParserMinimalBase.\_reportError(JsonParserMinimalBase.java:521) > at > org.elasticsearch.common.jackson.impl.JsonParserMinimalBase.\_throwUnquotedSpace(JsonParserMinimalBase.java:482) > at > org.elasticsearch.common.jackson.impl.Utf8StreamParser.\_finishString2(Utf8StreamParser.java:1975) > at > org.elasticsearch.common.jackson.impl.Utf8StreamParser.\_finishString(Utf8StreamParser.java:1905) > at > org.elasticsearch.common.jackson.impl.Utf8StreamParser.getText(Utf8StreamParser.java:276) > at > org.elasticsearch.common.xcontent.json.JsonXContentParser.text(JsonXContentParser.java:83) > at > org.elasticsearch.common.xcontent.support.AbstractXContentParser.textOrNull(AbstractXContentParser.java:106) > at > org.elasticsearch.index.mapper.core.StringFieldMapper.parseCreateField(StringFieldMapper.java:207) > at > org.elasticsearch.index.mapper.core.StringFieldMapper.parseCreateField(StringFieldMapper.java:42) > at > org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(AbstractFieldMapper.java:307) > ... 11 more > [2012-06-22 17:36:14,922][WARN][cluster.action.shard] [Cat-Man] > sending failed shard for [algotree][0], node[Rl9eKHxLRe6PDD9am-L9og], [P], > s[INITIALIZING], reason [Failed to start shard, message > [IndexShardGatewayRecoveryException[[algotree][0] failed to recover shard]; > nested: MapperParsingException[Failed to parse [Link]]; nested: > JsonParseException[Illegal unquoted character ((CTRL-CHAR, code 0)): has to > be escaped using backslash to be included in string value > at [Source: [B@11d565f; line: 1, column: 4062]]; ]] > [2012-06-22 17:36:14,922][WARN][cluster.action.shard] [Cat-Man] > received shard failed for [algotree][0], node[Rl9eKHxLRe6PDD9am-L9og], [P], > s[INITIALIZING], reason [Failed to start shard, message > [IndexShardGatewayRecoveryException[[algotree][0] failed to recover shard]; > nested: MapperParsingException[Failed to parse [Link]]; nested: > JsonParseException[Illegal unquoted character ((CTRL-CHAR, code 0)): has to > be escaped using backslash to be included in string value > at [Source: [B@11d565f; line: 1, column: 4062]]; ]] > [2012-06-22 17:36:17,165][WARN][indices.cluster] [Cat-Man] > [algotree][1] failed to start shard > org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: > [algotree][1] failed to recover shard > at > org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:201) > at > org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:177) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > Caused by: org.elasticsearch.index.mapper.MapperParsingException: Failed > to parse [Categories.Types.Facets.MoneyEvent.entity] > at > org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(AbstractFieldMapper.java:320) > at > org.elasticsearch.index.mapper.object.ObjectMapper.serializeValue(ObjectMapper.java:580) > at > org.elasticsearch.index.mapper.object.ObjectMapper.parse(ObjectMapper.java:444) > at > org.elasticsearch.index.mapper.object.ObjectMapper.serializeObject(ObjectMapper.java:492) > at > org.elasticsearch.index.mapper.object.ObjectMapper.serializeArray(ObjectMapper.java:560) > at > org.elasticsearch.index.mapper.object.ObjectMapper.parse(ObjectMapper.java:436) > at > org.elasticsearch.index.mapper.object.ObjectMapper.serializeObject(ObjectMapper.java:492) > at > org.elasticsearch.index.mapper.object.ObjectMapper.serializeArray(ObjectMapper.java:560) > at > org.elasticsearch.index.mapper.object.ObjectMapper.parse(ObjectMapper.java:436) > at > org.elasticsearch.index.mapper.object.ObjectMapper.serializeObject(ObjectMapper.java:492) > at > org.elasticsearch.index.mapper.object.ObjectMapper.parse(ObjectMapper.java:434) > at > org.elasticsearch.index.mapper.object.ObjectMapper.serializeObject(ObjectMapper.java:492) > at > org.elasticsearch.index.mapper.object.ObjectMapper.parse(ObjectMapper.java:434) > at > org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:473) > at > org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:417) > at > org.elasticsearch.index.shard.service.InternalIndexShard.prepareIndex(InternalIndexShard.java:311) > at > org.elasticsearch.index.shard.service.InternalIndexShard.performRecoveryOperation(InternalIndexShard.java:624) > at > org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:196) > ... 4 more > . > What could have posssibly happened? How can we resolve this issue.. we > are using Ubuntu..is there a way that i can get the data back.. > > Thanks in advance...... > > nel ---
**Author:** ![jagdeep](https://avatars.discourse-cdn.com/v4/letter/j/ba8739/32.png) [@jagdeep](https://discuss.elastic.co/u/jagdeep)\ **Post date:** [June 28, 2012, 10:17am UTC](https://discuss.elastic.co/t/es-failed-to-recover-after-crash/8195/3 "2012-06-28T10:17:06Z")
1. Please specify ES setup, number of nodes, shards etc. 2. Is it not able to recover after hard restart? On Jun 22, 6:36 pm, nel [NeilA...@algotree.com](mailto:NeilA...@algotree.com) wrote: > Hi > > We are facing an isssue with ES. Our ES instance crashed due to low disk > space. After that it triggers out exceptions which are described below.. > > at > org.elasticsearch.index.shard.service.InternalIndexShard.performRecoveryOpe ration(InternalIndexShard.java:624) > at > org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalI ndexShardGateway.java:196) > ... 4 more > Caused by: org.elasticsearch.common.jackson.JsonParseException: Illegal > unquoted character ((CTRL-CHAR, code 0)): has to be escaped using backslash > to be included in string value > at [Source: [B@bfc25c; line: 1, column: 4062] > at > org.elasticsearch.common.jackson.JsonParser.\_constructError(JsonParser.java :1432) > at > org.elasticsearch.common.jackson.impl.JsonParserMinimalBase.\_reportError(Js onParserMinimalBase.java:521) > at > org.elasticsearch.common.jackson.impl.JsonParserMinimalBase.\_throwUnquotedS pace(JsonParserMinimalBase.java:482) > at > org.elasticsearch.common.jackson.impl.Utf8StreamParser.\_finishString2(Utf8S treamParser.java:1975) > at > org.elasticsearch.common.jackson.impl.Utf8StreamParser.\_finishString(Utf8St reamParser.java:1905) > at > org.elasticsearch.common.jackson.impl.Utf8StreamParser.getText(Utf8StreamPa rser.java:276) > at > org.elasticsearch.common.xcontent.json.JsonXContentParser.text(JsonXContent Parser.java:83) > at > org.elasticsearch.common.xcontent.support.AbstractXContentParser.textOrNull (AbstractXContentParser.java:106) > at > org.elasticsearch.index.mapper.core.StringFieldMapper.parseCreateField(Stri ngFieldMapper.java:207) > at > org.elasticsearch.index.mapper.core.StringFieldMapper.parseCreateField(Stri ngFieldMapper.java:42) > at > org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(AbstractField Mapper.java:307) > ... 11 more > [2012-06-22 17:36:11,075][WARN][cluster.action.shard] [Cat-Man] > sending failed shard for [algotree][0], node[Rl9eKHxLRe6PDD9am-L9og], [P], > s[INITIALIZING], reason [Failed to start shard, message > [IndexShardGatewayRecoveryException[[algotree][0] failed to recover shard]; > nested: MapperParsingException[Failed to parse [Link]]; nested: > JsonParseException[Illegal unquoted character ((CTRL-CHAR, code 0)): has to > be escaped using backslash to be included in string value > at [Source: [B@bfc25c; line: 1, column: 4062]]; ]] > [2012-06-22 17:36:11,075][WARN][cluster.action.shard] [Cat-Man] > received shard failed for [algotree][0], node[Rl9eKHxLRe6PDD9am-L9og], [P], > s[INITIALIZING], reason [Failed to start shard, message > [IndexShardGatewayRecoveryException[[algotree][0] failed to recover shard]; > nested: MapperParsingException[Failed to parse [Link]]; nested: > JsonParseException[Illegal unquoted character ((CTRL-CHAR, code 0)): has to > be escaped using backslash to be included in string value > at [Source: [B@bfc25c; line: 1, column: 4062]]; ]] > [2012-06-22 17:36:13,107][WARN][indices.cluster] [Cat-Man] > [algotree][1] failed to start shard > org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: > [algotree][1] failed to recover shard > at > org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalI ndexShardGateway.java:201) > at > org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGa tewayService.java:177) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.j ava:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java: 908) > at java.lang.Thread.run(Thread.java:662) > Caused by: org.elasticsearch.index.mapper.MapperParsingException: Failed to > parse [Categories.Types.Facets.MoneyEvent.entity] > at > org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(AbstractField Mapper.java:320) > at > org.elasticsearch.index.mapper.object.ObjectMapper.serializeValue(ObjectMap per.java:580) > at > org.elasticsearch.index.mapper.object.ObjectMapper.parse(ObjectMapper.java: 444) > at > org.elasticsearch.index.mapper.object.ObjectMapper.serializeObject(ObjectMa pper.java:492) > at > org.elasticsearch.index.mapper.object.ObjectMapper.serializeArray(ObjectMap per.java:560) > at > org.elasticsearch.index.mapper.object.ObjectMapper.parse(ObjectMapper.java: 436) > at > org.elasticsearch.index.mapper.object.ObjectMapper.serializeObject(ObjectMa pper.java:492) > at > org.elasticsearch.index.mapper.object.ObjectMapper.serializeArray(ObjectMap per.java:560) > at > org.elasticsearch.index.mapper.object.ObjectMapper.parse(ObjectMapper.java: 436) > at > org.elasticsearch.index.mapper.object.ObjectMapper.serializeObject(ObjectMa pper.java:492) > at > org.elasticsearch.index.mapper.object.ObjectMapper.parse(ObjectMapper.java: 434) > at > org.elasticsearch.index.mapper.object.ObjectMapper.serializeObject(ObjectMa pper.java:492) > at > org.elasticsearch.index.mapper.object.ObjectMapper.parse(ObjectMapper.java: 434) > at > org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:473 ) > at > org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:417 ) > at > org.elasticsearch.index.shard.service.InternalIndexShard.prepareIndex(Inter nalIndexShard.java:311) > at > org.elasticsearch.index.shard.service.InternalIndexShard.performRecoveryOpe ration(InternalIndexShard.java:624) > at > org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalI ndexShardGateway.java:196) > ... 4 more > Caused by: org.elasticsearch.common.jackson.JsonParseException: Illegal > unquoted character ((CTRL-CHAR, code 0)): has to be escaped using backslash > to be included in string value > at [Source: [B@eb203b; line: 1, column: 937] > at > org.elasticsearch.common.jackson.JsonParser.\_constructError(JsonParser.java :1432) > at > org.elasticsearch.common.jackson.impl.JsonParserMinimalBase.\_reportError(Js onParserMinimalBase.java:521) > at > org.elasticsearch.common.jackson.impl.JsonParserMinimalBase.\_throwUnquotedS pace(JsonParserMinimalBase.java:482) > at > org.elasticsearch.common.jackson.impl.Utf8StreamParser.\_finishString2(Utf8S treamParser.java:1975) > at > org.elasticsearch.common.jackson.impl.Utf8StreamParser.\_finishString(Utf8St reamParser.java:1905) > at > org.elasticsearch.common.jackson.impl.Utf8StreamParser.getText(Utf8StreamPa rser.java:276) > at > org.elasticsearch.common.xcontent.json.JsonXContentParser.text(JsonXContent Parser.java:83) > at > org.elasticsearch.common.xcontent.support.AbstractXContentParser.textOrNull (AbstractXContentParser.java:106) > at > org.elasticsearch.index.mapper.core.StringFieldMapper.parseCreateField(Stri ngFieldMapper.java:207) > at > org.elasticsearch.index.mapper.core.StringFieldMapper.parseCreateField(Stri ngFieldMapper.java:42) > at > org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(AbstractField Mapper.java:307) > ... 21 more > [2012-06-22 17:36:13,135][WARN][cluster.action.shard] [Cat-Man] > sending failed shard for [algotree][1], node[Rl9eKHxLRe6PDD9am-L9og], [P], > s[INITIALIZING], reason [Failed to start shard, message > [IndexShardGatewayRecoveryException[[algotree][1] failed to recover shard]; > nested: MapperParsingException[Failed to parse > [Categories.Types.Facets.MoneyEvent.entity]]; nested: > JsonParseException[Illegal unquoted character ((CTRL-CHAR, code 0)): has to > be escaped using backslash to be included in string value > at [Source: [B@eb203b; line: 1, column: 937]]; ]] > [2012-06-22 17:36:13,135][WARN][cluster.action.shard] [Cat-Man] > received shard failed for [algotree][1], node[Rl9eKHxLRe6PDD9am-L9og], [P], > s[INITIALIZING], reason [Failed to start shard, message > [IndexShardGatewayRecoveryException[[algotree][1] failed to recover shard]; > nested: MapperParsingException[Failed to parse > [Categories.Types.Facets.MoneyEvent.entity]]; nested: > JsonParseException[Illegal unquoted character ((CTRL-CHAR, code 0)): has to > be escaped using backslash to be included in string value > at [Source: [B@eb203b; line: 1, column: 937]]; ]] > [2012-06-22 17:36:14,895][WARN][indices.cluster] [Cat-Man] > [algotree][0] failed to start shard > org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: > [algotree][0] failed to recover shard > at > org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalI ndexShardGateway.java:201) > at > org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGa tewayService.java:177) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.j ava:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java: 908) > at java.lang.Thread.run(Thread.java:662) > Caused by: org.elasticsearch.index.mapper.MapperParsingException: Failed to > parse [Link] > at > org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(AbstractField Mapper.java:320) > at > org.elasticsearch.index.mapper.object.ObjectMapper.serializeValue(ObjectMap per.java:580) > at > org.elasticsearch.index.mapper.object.ObjectMapper.parse(ObjectMapper.java: 444) > at > org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:473 ) > at > org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:417 ) > at > org.elasticsearch.index.shard.service.InternalIndexShard.prepareIndex(Inter nalIndexShard.java:311) > at > org.elasticsearch.index.shard.service.InternalIndexShard.performRecoveryOpe ration(InternalIndexShard.java:624) > at > org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalI ndexShardGateway.java:196) > ... 4 more > Caused by: org.elasticsearch.common.jackson.JsonParseException: Illegal > unquoted character ((CTRL-CHAR, code 0)): has to be escaped using backslash > to be included in string value > at [Source: [B@11d565f; line: 1, column: 4062] > at > org.elasticsearch.common.jackson.JsonParser.\_constructError(JsonParser.java :1432) > at > org.elasticsearch.common.jackson.impl.JsonParserMinimalBase.\_reportError(Js onParserMinimalBase.java:521) > at > ... > > read more ยป ---
**Author:** ![Clinton\_Gormley](https://avatars.discourse-cdn.com/v4/letter/c/50afbb/32.png) [@Clinton\_Gormley](https://discuss.elastic.co/u/Clinton_Gormley)\ **Post date:** [June 28, 2012, 10:40am UTC](https://discuss.elastic.co/t/es-failed-to-recover-after-crash/8195/4 "2012-06-28T10:40:06Z")
Hiya > We are facing an isssue with ES. Our ES instance crashed due to low > disk space. After that it triggers out exceptions which are described > below.. > > shard]; nested: MapperParsingException[Failed to parse [Link]]; > nested: JsonParseException[Illegal unquoted character ((CTRL-CHAR, > code 0)): has to be escaped using backslash to be included in string > value I'm guessing here, but it looks like your mapping has been corrupted, presumably by running out of disk space. You _may_ be able to fix it by looking in data/nodes/0/indices/INDEXNAME/\_state. But that file is compressed, I'm not sure what format. Do you have an older copy that you might be able to use? And it sounds a lot like you were only running one instance, so no other instance you could copy the mapping from clint ---
**Author:** ![Igor\_Motov](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/igor_motov/32/45193_2.png) [@Igor\_Motov](https://discuss.elastic.co/u/Igor_Motov)\ **Post date:** [June 28, 2012, 3:16pm UTC](https://discuss.elastic.co/t/es-failed-to-recover-after-crash/8195/5 "2012-06-28T15:16:23Z")
In nel's case it was corrupted transaction log. When you run out of disk space sometimes the last transaction cannot be fully written into transaction log and then it fails on recovery. If you see exactly the same error messages, you can try the following: - shut down elasticsearch cluster - find all shards that cannot recover by searching log file - for each shard move its non-zero length translog file into a temporary directory (see explanation below) - start elasticsearch cluster - if you see messages for other shards - repeat If you see message like this: [2012-06-22 17:36:17,165][WARN][indices.cluster] [Cat-Man] [\* myindex\*][_1_] failed to start shard It means that it cannot recover shard _1_ of the index _myindex_ on the node Cat-Man. If you take a look at data/elasticsearch/nodes/0/indices/\* myindex/1\*/translog directory, you will find files like this: translog-123456677899 or translog-123456677899.recovering. One of them will have non-zero length. Move it to a temporary directory and try starting the server. The transaction log files that you will be moving out contain your most recently updated and indexed documents. So, these updates will be lost as a result of this operations, but you should be able to recover the rest of your data. On Thursday, June 28, 2012 6:40:06 AM UTC-4, Clinton Gormley wrote: > Hiya > > > We are facing an isssue with ES. Our ES instance crashed due to low > > disk space. After that it triggers out exceptions which are described > > below.. > > > > shard]; nested: MapperParsingException[Failed to parse [Link]]; > > nested: JsonParseException[Illegal unquoted character ((CTRL-CHAR, > > code 0)): has to be escaped using backslash to be included in string > > value > > I'm guessing here, but it looks like your mapping has been corrupted, > presumably by running out of disk space. > > You _may_ be able to fix it by looking in > data/nodes/0/indices/INDEXNAME/\_state. But that file is compressed, I'm > not sure what format. Do you have an older copy that you might be able > to use? > > And it sounds a lot like you were only running one instance, so no other > instance you could copy the mapping from > > clint ---
**Author:** ![Laurent\_Block](https://avatars.discourse-cdn.com/v4/letter/l/74df32/32.png) [@Laurent\_Block](https://discuss.elastic.co/u/Laurent_Block)\ **Post date:** [July 4, 2012, 9:35am UTC](https://discuss.elastic.co/t/es-failed-to-recover-after-crash/8195/6 "2012-07-04T09:35:39Z")
Hi there, We have a similar problem with our graylog2 constellation. We have 2 nodes in master-master, during the weekend (regarding a huge processing of files) we also ran out of space on the harddisks. We enlarged the partitions in order to get again some free space on the disks. Since then, we have a CRITICAL error on our monitoring system which says that string "green" cannot be found on the health status report of the cluster, here's the curl output from the server: { "cluster\_name" : "graylog2", "status" : "red", "timed\_out" : false, "number\_of\_nodes" : 2, "number\_of\_data\_nodes" : 2, "active\_primary\_shards" : 1, "active\_shards" : 2, "relocating\_shards" : 0, "initializing\_shards" : 1, "unassigned\_shards" : 1 } As we have 1 unassigned shard, I run deeper in debugging and found this in the logs: Server1: [WARN][cluster.action.shard] [logserver01] received shard failed for [graylog2][0], node[B3AFfzhhQ8yMJmuaJF6v6g], [P], s[INITIALIZING], reason [Failed to start shard, message [IndexShardGatewayRecoveryException[[graylog2][0] failed to recover shard]; nested: MapperParsingException[Failed to parse [full\_message]]; nested: JsonParseException[Illegal unquoted character ((CTRL-CHAR, code 0)): has to be escaped using backslash to be included in string value at [Source: [B@4bd8d548; line: 1, column: 575]]; ]] Server2: [WARN][cluster.action.shard] [logserver02] sending failed shard for [graylog2][0], node[lNCk9JoQTIKrT76k2B1pUw], [P], s[INITIALIZING], reason [Failed to start shard, message [IndexShardGatewayRecoveryException[[graylog2][0] failed to recover shard]; nested: MapperParsingException[Failed to parse [message]]; nested: JsonParseException[Illegal unquoted character ((CTRL-CHAR, code 0)): has to be escaped using backslash to be included in string value at [Source: [B@77a202bf; line: 1, column: 79]]; ]] Regarding the last comment that Igor wrote, do I only have to do the procedure he described on the second server where the "sending" failed, or on the whole cluster? I'm really a newbie on elasticsearch so sorry if I cannot provide you more professional input. On Thursday, June 28, 2012 5:16:23 PM UTC+2, Igor Motov wrote: > In nel's case it was corrupted transaction log. When you run out of disk > space sometimes the last transaction cannot be fully written into > transaction log and then it fails on recovery. If you see exactly the same > error messages, you can try the following: > > - shut down elasticsearch cluster > - find all shards that cannot recover by searching log file > - for each shard move its non-zero length translog file into a temporary > directory (see explanation below) > - start elasticsearch cluster > - if you see messages for other shards - repeat > > If you see message like this: > > [2012-06-22 17:36:17,165][WARN][indices.cluster] [Cat-Man] [\* > myindex\*][_1_] failed to start shard > > It means that it cannot recover shard _1_ of the index _myindex_ on the > node Cat-Man. If you take a look at data/elasticsearch/nodes/0/indices/\* > myindex/1\*/translog directory, you will find files like this: > translog-123456677899 or translog-123456677899.recovering. One of them will > have non-zero length. Move it to a temporary directory and try starting the > server. > > The transaction log files that you will be moving out contain your most > recently updated and indexed documents. So, these updates will be lost as a > result of this operations, but you should be able to recover the rest of > your data. > > On Thursday, June 28, 2012 6:40:06 AM UTC-4, Clinton Gormley wrote: > > > Hiya > > > > > We are facing an isssue with ES. Our ES instance crashed due to low > > > disk space. After that it triggers out exceptions which are described > > > below.. > > > > > > shard]; nested: MapperParsingException[Failed to parse [Link]]; > > > nested: JsonParseException[Illegal unquoted character ((CTRL-CHAR, > > > code 0)): has to be escaped using backslash to be included in string > > > value > > > > I'm guessing here, but it looks like your mapping has been corrupted, > > presumably by running out of disk space. > > > > You _may_ be able to fix it by looking in > > data/nodes/0/indices/INDEXNAME/\_state. But that file is compressed, I'm > > not sure what format. Do you have an older copy that you might be able > > to use? > > > > And it sounds a lot like you were only running one instance, so no other > > instance you could copy the mapping from > > > > clint ---
**Author:** ![Laurent\_Block](https://avatars.discourse-cdn.com/v4/letter/l/74df32/32.png) [@Laurent\_Block](https://discuss.elastic.co/u/Laurent_Block)\ **Post date:** [July 4, 2012, 1:06pm UTC](https://discuss.elastic.co/t/es-failed-to-recover-after-crash/8195/7 "2012-07-04T13:06:31Z")
A more explicit output from the cluster health here: { "cluster\_name" : "graylog2", "status" : "red", "timed\_out" : false, "number\_of\_nodes" : 2, "number\_of\_data\_nodes" : 2, "active\_primary\_shards" : 1, "active\_shards" : 2, "relocating\_shards" : 0, "initializing\_shards" : 1, "unassigned\_shards" : 1, "indices" : { "graylog2" : { "status" : "red", "number\_of\_shards" : 2, "number\_of\_replicas" : 1, "active\_primary\_shards" : 1, "active\_shards" : 2, "relocating\_shards" : 0, "initializing\_shards" : 1, "unassigned\_shards" : 1, "shards" : { "0" : { "status" : "red", "primary\_active" : false, "active\_shards" : 0, "relocating\_shards" : 0, "initializing\_shards" : 1, "unassigned\_shards" : 1 }, "1" : { "status" : "green", "primary\_active" : true, "active\_shards" : 2, "relocating\_shards" : 0, "initializing\_shards" : 0, "unassigned\_shards" : 0 } } } } } Now at least I know there the problem is, on shard 0 of my graylog2 cluster. Here's an extract of the file system (translog folder): Server1: -rw-rw-r-- 1 elasticsearch elasticsearch 323580 Jul 4 15:05 translog-1337094255321 -rw-rw-r-- 1 elasticsearch elasticsearch 1142760 Jun 29 18:33 translog-1337094255321.recovering Server2: total 516 -rw-rw-r-- 1 elasticsearch elasticsearch 0 Jul 4 15:05 translog-1337094255544 -rw-rw-r-- 1 elasticsearch elasticsearch 524288 Jul 2 10:08 translog-1337094255544.recovering Any advice what I can do now? Many thanks for your help in advance. phoenix On Wednesday, July 4, 2012 11:35:39 AM UTC+2, phoenix2lux wrote: > Hi there, > > We have a similar problem with our graylog2 constellation. > We have 2 nodes in master-master, during the weekend (regarding a huge > processing of files) we also ran out of space on the harddisks. We enlarged > the partitions in order to get again some free space on the disks. > Since then, we have a CRITICAL error on our monitoring system which says > that string "green" cannot be found on the health status report of the > cluster, here's the curl output from the server: > > { > "cluster\_name" : "graylog2", > "status" : "red", > "timed\_out" : false, > "number\_of\_nodes" : 2, > "number\_of\_data\_nodes" : 2, > "active\_primary\_shards" : 1, > "active\_shards" : 2, > "relocating\_shards" : 0, > "initializing\_shards" : 1, > "unassigned\_shards" : 1 > } > > As we have 1 unassigned shard, I run deeper in debugging and found this in > the logs: > > Server1: > > [WARN][cluster.action.shard] [logserver01] received shard failed for > [graylog2][0], node[B3AFfzhhQ8yMJmuaJF6v6g], [P], s[INITIALIZING], reason > [Failed to start shard, message > [IndexShardGatewayRecoveryException[[graylog2][0] failed to recover shard]; > nested: MapperParsingException[Failed to parse [full\_message]]; nested: > JsonParseException[Illegal unquoted character ((CTRL-CHAR, code 0)): has to > be escaped using backslash to be included in string value at [Source: > [B@4bd8d548; line: 1, column: 575]]; ]] > > Server2: > > [WARN][cluster.action.shard] [logserver02] sending failed shard for > [graylog2][0], node[lNCk9JoQTIKrT76k2B1pUw], [P], s[INITIALIZING], reason > [Failed to start shard, message > [IndexShardGatewayRecoveryException[[graylog2][0] failed to recover shard]; > nested: MapperParsingException[Failed to parse [message]]; nested: > JsonParseException[Illegal unquoted character ((CTRL-CHAR, code 0)): has to > be escaped using backslash to be included in string value at [Source: > [B@77a202bf; line: 1, column: 79]]; ]] > > Regarding the last comment that Igor wrote, do I only have to do the > procedure he described on the second server where the "sending" failed, or > on the whole cluster? > > I'm really a newbie on elasticsearch so sorry if I cannot provide you more > professional input. ---
**Author:** ![Igor\_Motov](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/igor_motov/32/45193_2.png) [@Igor\_Motov](https://discuss.elastic.co/u/Igor_Motov)\ **Post date:** [July 8, 2012, 3:19pm UTC](https://discuss.elastic.co/t/es-failed-to-recover-after-crash/8195/8 "2012-07-08T15:19:14Z")
Hi phoenix, First of all I would suggest to stop all indexing until cluster is recovered. Then I would start with removing all translog-_.recovering files on all shards that report issues in the log and if this wouldn't help - removing all non-zero length translog-_ files on these shards. Igor On Wednesday, July 4, 2012 9:06:31 AM UTC-4, phoenix2lux wrote: > A more explicit output from the cluster health here: > > { > "cluster\_name" : "graylog2", > "status" : "red", > "timed\_out" : false, > "number\_of\_nodes" : 2, > "number\_of\_data\_nodes" : 2, > "active\_primary\_shards" : 1, > "active\_shards" : 2, > "relocating\_shards" : 0, > "initializing\_shards" : 1, > "unassigned\_shards" : 1, > "indices" : { > "graylog2" : { > "status" : "red", > "number\_of\_shards" : 2, > "number\_of\_replicas" : 1, > "active\_primary\_shards" : 1, > "active\_shards" : 2, > "relocating\_shards" : 0, > "initializing\_shards" : 1, > "unassigned\_shards" : 1, > "shards" : { > "0" : { > "status" : "red", > "primary\_active" : false, > "active\_shards" : 0, > "relocating\_shards" : 0, > "initializing\_shards" : 1, > "unassigned\_shards" : 1 > }, > "1" : { > "status" : "green", > "primary\_active" : true, > "active\_shards" : 2, > "relocating\_shards" : 0, > "initializing\_shards" : 0, > "unassigned\_shards" : 0 > } > } > } > } > } > > Now at least I know there the problem is, on shard 0 of my graylog2 > cluster. > Here's an extract of the file system (translog folder): > > Server1: > -rw-rw-r-- 1 elasticsearch elasticsearch 323580 Jul 4 15:05 > translog-1337094255321 > -rw-rw-r-- 1 elasticsearch elasticsearch 1142760 Jun 29 18:33 > translog-1337094255321.recovering > > Server2: > total 516 > -rw-rw-r-- 1 elasticsearch elasticsearch 0 Jul 4 15:05 > translog-1337094255544 > -rw-rw-r-- 1 elasticsearch elasticsearch 524288 Jul 2 10:08 > translog-1337094255544.recovering > > Any advice what I can do now? > Many thanks for your help in advance. > > phoenix > > On Wednesday, July 4, 2012 11:35:39 AM UTC+2, phoenix2lux wrote: > > > Hi there, > > > > We have a similar problem with our graylog2 constellation. > > We have 2 nodes in master-master, during the weekend (regarding a huge > > processing of files) we also ran out of space on the harddisks. We enlarged > > the partitions in order to get again some free space on the disks. > > Since then, we have a CRITICAL error on our monitoring system which says > > that string "green" cannot be found on the health status report of the > > cluster, here's the curl output from the server: > > > > { > > "cluster\_name" : "graylog2", > > "status" : "red", > > "timed\_out" : false, > > "number\_of\_nodes" : 2, > > "number\_of\_data\_nodes" : 2, > > "active\_primary\_shards" : 1, > > "active\_shards" : 2, > > "relocating\_shards" : 0, > > "initializing\_shards" : 1, > > "unassigned\_shards" : 1 > > } > > > > As we have 1 unassigned shard, I run deeper in debugging and found this > > in the logs: > > > > Server1: > > > > [WARN][cluster.action.shard] [logserver01] received shard failed > > for [graylog2][0], node[B3AFfzhhQ8yMJmuaJF6v6g], [P], s[INITIALIZING], > > reason [Failed to start shard, message > > [IndexShardGatewayRecoveryException[[graylog2][0] failed to recover shard]; > > nested: MapperParsingException[Failed to parse [full\_message]]; nested: > > JsonParseException[Illegal unquoted character ((CTRL-CHAR, code 0)): has to > > be escaped using backslash to be included in string value at [Source: > > [B@4bd8d548; line: 1, column: 575]]; ]] > > > > Server2: > > > > [WARN][cluster.action.shard] [logserver02] sending failed shard for > > [graylog2][0], node[lNCk9JoQTIKrT76k2B1pUw], [P], s[INITIALIZING], reason > > [Failed to start shard, message > > [IndexShardGatewayRecoveryException[[graylog2][0] failed to recover shard]; > > nested: MapperParsingException[Failed to parse [message]]; nested: > > JsonParseException[Illegal unquoted character ((CTRL-CHAR, code 0)): has to > > be escaped using backslash to be included in string value at [Source: > > [B@77a202bf; line: 1, column: 79]]; ]] > > > > Regarding the last comment that Igor wrote, do I only have to do the > > procedure he described on the second server where the "sending" failed, or > > on the whole cluster? > > > > I'm really a newbie on elasticsearch so sorry if I cannot provide you > > more professional input. ---
**Author:** ![Laurent\_Block](https://avatars.discourse-cdn.com/v4/letter/l/74df32/32.png) [@Laurent\_Block](https://discuss.elastic.co/u/Laurent_Block)\ **Post date:** [July 8, 2012, 3:50pm UTC](https://discuss.elastic.co/t/es-failed-to-recover-after-crash/8195/9 "2012-07-08T15:50:36Z")
Hi Igor, I followed all these steps and after waiting couple of minutes for the re-indexing of the ES cluster, the health of the cluster went green again and all shards are active and synced. Many thanks for your help! On 08.07.2012, at 17:19, Igor Motov [imotov@gmail.com](mailto:imotov@gmail.com) wrote: > Hi phoenix, > > First of all I would suggest to stop all indexing until cluster is recovered. Then I would start with removing all translog-_.recovering files on all shards that report issues in the log and if this wouldn't help - removing all non-zero length translog-_ files on these shards. > > Igor > > On Wednesday, July 4, 2012 9:06:31 AM UTC-4, phoenix2lux wrote: > > > A more explicit output from the cluster health here: > > > > { > > "cluster\_name" : "graylog2", > > "status" : "red", > > "timed\_out" : false, > > "number\_of\_nodes" : 2, > > "number\_of\_data\_nodes" : 2, > > "active\_primary\_shards" : 1, > > "active\_shards" : 2, > > "relocating\_shards" : 0, > > "initializing\_shards" : 1, > > "unassigned\_shards" : 1, > > "indices" : { > > "graylog2" : { > > "status" : "red", > > "number\_of\_shards" : 2, > > "number\_of\_replicas" : 1, > > "active\_primary\_shards" : 1, > > "active\_shards" : 2, > > "relocating\_shards" : 0, > > "initializing\_shards" : 1, > > "unassigned\_shards" : 1, > > "shards" : { > > "0" : { > > "status" : "red", > > "primary\_active" : false, > > "active\_shards" : 0, > > "relocating\_shards" : 0, > > "initializing\_shards" : 1, > > "unassigned\_shards" : 1 > > }, > > "1" : { > > "status" : "green", > > "primary\_active" : true, > > "active\_shards" : 2, > > "relocating\_shards" : 0, > > "initializing\_shards" : 0, > > "unassigned\_shards" : 0 > > } > > } > > } > > } > > } > > > > Now at least I know there the problem is, on shard 0 of my graylog2 cluster. > > Here's an extract of the file system (translog folder): > > > > Server1: > > -rw-rw-r-- 1 elasticsearch elasticsearch 323580 Jul 4 15:05 translog-1337094255321 > > -rw-rw-r-- 1 elasticsearch elasticsearch 1142760 Jun 29 18:33 translog-1337094255321.recovering > > > > Server2: > > total 516 > > -rw-rw-r-- 1 elasticsearch elasticsearch 0 Jul 4 15:05 translog-1337094255544 > > -rw-rw-r-- 1 elasticsearch elasticsearch 524288 Jul 2 10:08 translog-1337094255544.recovering > > > > Any advice what I can do now? > > Many thanks for your help in advance. > > > > phoenix > > > > On Wednesday, July 4, 2012 11:35:39 AM UTC+2, phoenix2lux wrote: > > > > > Hi there, > > > > > > We have a similar problem with our graylog2 constellation. > > > We have 2 nodes in master-master, during the weekend (regarding a huge processing of files) we also ran out of space on the harddisks. We enlarged the partitions in order to get again some free space on the disks. > > > Since then, we have a CRITICAL error on our monitoring system which says that string "green" cannot be found on the health status report of the cluster, here's the curl output from the server: > > > > > > { > > > "cluster\_name" : "graylog2", > > > "status" : "red", > > > "timed\_out" : false, > > > "number\_of\_nodes" : 2, > > > "number\_of\_data\_nodes" : 2, > > > "active\_primary\_shards" : 1, > > > "active\_shards" : 2, > > > "relocating\_shards" : 0, > > > "initializing\_shards" : 1, > > > "unassigned\_shards" : 1 > > > } > > > > > > As we have 1 unassigned shard, I run deeper in debugging and found this in the logs: > > > > > > Server1: > > > [WARN][cluster.action.shard] [logserver01] received shard failed for [graylog2][0], node[B3AFfzhhQ8yMJmuaJF6v6g], [P], s[INITIALIZING], reason [Failed to start shard, message [IndexShardGatewayRecoveryException[[graylog2][0] failed to recover shard]; nested: MapperParsingException[Failed to parse [full\_message]]; nested: JsonParseException[Illegal unquoted character ((CTRL-CHAR, code 0)): has to be escaped using backslash to be included in string value at [Source: [B@4bd8d548; line: 1, column: 575]]; ]] > > > > > > Server2: > > > [WARN][cluster.action.shard] [logserver02] sending failed shard for [graylog2][0], node[lNCk9JoQTIKrT76k2B1pUw], [P], s[INITIALIZING], reason [Failed to start shard, message [IndexShardGatewayRecoveryException[[graylog2][0] failed to recover shard]; nested: MapperParsingException[Failed to parse [message]]; nested: JsonParseException[Illegal unquoted character ((CTRL-CHAR, code 0)): has to be escaped using backslash to be included in string value at [Source: [B@77a202bf; line: 1, column: 79]]; ]] > > > > > > Regarding the last comment that Igor wrote, do I only have to do the procedure he described on the second server where the "sending" failed, or on the whole cluster? > > > > > > I'm really a newbie on elasticsearch so sorry if I cannot provide you more professional input. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 3:21am UTC](https://discuss.elastic.co/t/es-failed-to-recover-after-crash/8195/10 "2017-07-06T03:21:06Z")