ES failed to recover after crash

nelalx · June 22, 2012, 1:36pm

Hi

We are facing an isssue with ES. Our ES instance crashed due to low disk
space. After that it triggers out exceptions which are described below..

at
org.elasticsearch.index.shard.service.InternalIndexShard.performRecoveryOperation(InternalIndexShard.java:624)
at
org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:196)
... 4 more
Caused by: org.elasticsearch.common.jackson.JsonParseException: Illegal
unquoted character ((CTRL-CHAR, code 0)): has to be escaped using backslash
to be included in string value
at [Source: [B@bfc25c; line: 1, column: 4062]
at
org.elasticsearch.common.jackson.JsonParser._constructError(JsonParser.java:1432)
at
org.elasticsearch.common.jackson.impl.JsonParserMinimalBase._reportError(JsonParserMinimalBase.java:521)
at
org.elasticsearch.common.jackson.impl.JsonParserMinimalBase._throwUnquotedSpace(JsonParserMinimalBase.java:482)
at
org.elasticsearch.common.jackson.impl.Utf8StreamParser._finishString2(Utf8StreamParser.java:1975)
at
org.elasticsearch.common.jackson.impl.Utf8StreamParser._finishString(Utf8StreamParser.java:1905)
at
org.elasticsearch.common.jackson.impl.Utf8StreamParser.getText(Utf8StreamParser.java:276)
at
org.elasticsearch.common.xcontent.json.JsonXContentParser.text(JsonXContentParser.java:83)
at
org.elasticsearch.common.xcontent.support.AbstractXContentParser.textOrNull(AbstractXContentParser.java:106)
at
org.elasticsearch.index.mapper.core.StringFieldMapper.parseCreateField(StringFieldMapper.java:207)
at
org.elasticsearch.index.mapper.core.StringFieldMapper.parseCreateField(StringFieldMapper.java:42)
at
org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(AbstractFieldMapper.java:307)
... 11 more
[2012-06-22 17:36:11,075][WARN ][cluster.action.shard ] [Cat-Man]
sending failed shard for [algotree][0], node[Rl9eKHxLRe6PDD9am-L9og], [P],
s[INITIALIZING], reason [Failed to start shard, message
[IndexShardGatewayRecoveryException[[algotree][0] failed to recover shard];
nested: MapperParsingException[Failed to parse [Link]]; nested:
JsonParseException[Illegal unquoted character ((CTRL-CHAR, code 0)): has to
be escaped using backslash to be included in string value
at [Source: [B@bfc25c; line: 1, column: 4062]]; ]]
[2012-06-22 17:36:11,075][WARN ][cluster.action.shard ] [Cat-Man]
received shard failed for [algotree][0], node[Rl9eKHxLRe6PDD9am-L9og], [P],
s[INITIALIZING], reason [Failed to start shard, message
[IndexShardGatewayRecoveryException[[algotree][0] failed to recover shard];
nested: MapperParsingException[Failed to parse [Link]]; nested:
JsonParseException[Illegal unquoted character ((CTRL-CHAR, code 0)): has to
be escaped using backslash to be included in string value
at [Source: [B@bfc25c; line: 1, column: 4062]]; ]]
[2012-06-22 17:36:13,107][WARN ][indices.cluster ] [Cat-Man]
[algotree][1] failed to start shard
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException:
[algotree][1] failed to recover shard
at
org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:201)
at
org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:177)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: org.elasticsearch.index.mapper.MapperParsingException: Failed to
parse [Categories.Types.Facets.MoneyEvent.entity]
at
org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(AbstractFieldMapper.java:320)
at
org.elasticsearch.index.mapper.object.ObjectMapper.serializeValue(ObjectMapper.java:580)
at
org.elasticsearch.index.mapper.object.ObjectMapper.parse(ObjectMapper.java:444)
at
org.elasticsearch.index.mapper.object.ObjectMapper.serializeObject(ObjectMapper.java:492)
at
org.elasticsearch.index.mapper.object.ObjectMapper.serializeArray(ObjectMapper.java:560)
at
org.elasticsearch.index.mapper.object.ObjectMapper.parse(ObjectMapper.java:436)
at
org.elasticsearch.index.mapper.object.ObjectMapper.serializeObject(ObjectMapper.java:492)
at
org.elasticsearch.index.mapper.object.ObjectMapper.serializeArray(ObjectMapper.java:560)
at
org.elasticsearch.index.mapper.object.ObjectMapper.parse(ObjectMapper.java:436)
at
org.elasticsearch.index.mapper.object.ObjectMapper.serializeObject(ObjectMapper.java:492)
at
org.elasticsearch.index.mapper.object.ObjectMapper.parse(ObjectMapper.java:434)
at
org.elasticsearch.index.mapper.object.ObjectMapper.serializeObject(ObjectMapper.java:492)
at
org.elasticsearch.index.mapper.object.ObjectMapper.parse(ObjectMapper.java:434)
at
org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:473)
at
org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:417)
at
org.elasticsearch.index.shard.service.InternalIndexShard.prepareIndex(InternalIndexShard.java:311)
at
org.elasticsearch.index.shard.service.InternalIndexShard.performRecoveryOperation(InternalIndexShard.java:624)
at
org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:196)
... 4 more
Caused by: org.elasticsearch.common.jackson.JsonParseException: Illegal
unquoted character ((CTRL-CHAR, code 0)): has to be escaped using backslash
to be included in string value
at [Source: [B@eb203b; line: 1, column: 937]
at
org.elasticsearch.common.jackson.JsonParser._constructError(JsonParser.java:1432)
at
org.elasticsearch.common.jackson.impl.JsonParserMinimalBase._reportError(JsonParserMinimalBase.java:521)
at
org.elasticsearch.common.jackson.impl.JsonParserMinimalBase._throwUnquotedSpace(JsonParserMinimalBase.java:482)
at
org.elasticsearch.common.jackson.impl.Utf8StreamParser._finishString2(Utf8StreamParser.java:1975)
at
org.elasticsearch.common.jackson.impl.Utf8StreamParser._finishString(Utf8StreamParser.java:1905)
at
org.elasticsearch.common.jackson.impl.Utf8StreamParser.getText(Utf8StreamParser.java:276)
at
org.elasticsearch.common.xcontent.json.JsonXContentParser.text(JsonXContentParser.java:83)
at
org.elasticsearch.common.xcontent.support.AbstractXContentParser.textOrNull(AbstractXContentParser.java:106)
at
org.elasticsearch.index.mapper.core.StringFieldMapper.parseCreateField(StringFieldMapper.java:207)
at
org.elasticsearch.index.mapper.core.StringFieldMapper.parseCreateField(StringFieldMapper.java:42)
at
org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(AbstractFieldMapper.java:307)
... 21 more
[2012-06-22 17:36:13,135][WARN ][cluster.action.shard ] [Cat-Man]
sending failed shard for [algotree][1], node[Rl9eKHxLRe6PDD9am-L9og], [P],
s[INITIALIZING], reason [Failed to start shard, message
[IndexShardGatewayRecoveryException[[algotree][1] failed to recover shard];
nested: MapperParsingException[Failed to parse
[Categories.Types.Facets.MoneyEvent.entity]]; nested:
JsonParseException[Illegal unquoted character ((CTRL-CHAR, code 0)): has to
be escaped using backslash to be included in string value
at [Source: [B@eb203b; line: 1, column: 937]]; ]]
[2012-06-22 17:36:13,135][WARN ][cluster.action.shard ] [Cat-Man]
received shard failed for [algotree][1], node[Rl9eKHxLRe6PDD9am-L9og], [P],
s[INITIALIZING], reason [Failed to start shard, message
[IndexShardGatewayRecoveryException[[algotree][1] failed to recover shard];
nested: MapperParsingException[Failed to parse
[Categories.Types.Facets.MoneyEvent.entity]]; nested:
JsonParseException[Illegal unquoted character ((CTRL-CHAR, code 0)): has to
be escaped using backslash to be included in string value
at [Source: [B@eb203b; line: 1, column: 937]]; ]]
[2012-06-22 17:36:14,895][WARN ][indices.cluster ] [Cat-Man]
[algotree][0] failed to start shard
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException:
[algotree][0] failed to recover shard
at
org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:201)
at
org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:177)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: org.elasticsearch.index.mapper.MapperParsingException: Failed to
parse [Link]
at
org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(AbstractFieldMapper.java:320)
at
org.elasticsearch.index.mapper.object.ObjectMapper.serializeValue(ObjectMapper.java:580)
at
org.elasticsearch.index.mapper.object.ObjectMapper.parse(ObjectMapper.java:444)
at
org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:473)
at
org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:417)
at
org.elasticsearch.index.shard.service.InternalIndexShard.prepareIndex(InternalIndexShard.java:311)
at
org.elasticsearch.index.shard.service.InternalIndexShard.performRecoveryOperation(InternalIndexShard.java:624)
at
org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:196)
... 4 more
Caused by: org.elasticsearch.common.jackson.JsonParseException: Illegal
unquoted character ((CTRL-CHAR, code 0)): has to be escaped using backslash
to be included in string value
at [Source: [B@11d565f; line: 1, column: 4062]
at
org.elasticsearch.common.jackson.JsonParser._constructError(JsonParser.java:1432)
at
org.elasticsearch.common.jackson.impl.JsonParserMinimalBase._reportError(JsonParserMinimalBase.java:521)
at
org.elasticsearch.common.jackson.impl.JsonParserMinimalBase._throwUnquotedSpace(JsonParserMinimalBase.java:482)
at
org.elasticsearch.common.jackson.impl.Utf8StreamParser._finishString2(Utf8StreamParser.java:1975)
at
org.elasticsearch.common.jackson.impl.Utf8StreamParser._finishString(Utf8StreamParser.java:1905)
at
org.elasticsearch.common.jackson.impl.Utf8StreamParser.getText(Utf8StreamParser.java:276)
at
org.elasticsearch.common.xcontent.json.JsonXContentParser.text(JsonXContentParser.java:83)
at
org.elasticsearch.common.xcontent.support.AbstractXContentParser.textOrNull(AbstractXContentParser.java:106)
at
org.elasticsearch.index.mapper.core.StringFieldMapper.parseCreateField(StringFieldMapper.java:207)
at
org.elasticsearch.index.mapper.core.StringFieldMapper.parseCreateField(StringFieldMapper.java:42)
at
org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(AbstractFieldMapper.java:307)
... 11 more
[2012-06-22 17:36:14,922][WARN ][cluster.action.shard ] [Cat-Man]
sending failed shard for [algotree][0], node[Rl9eKHxLRe6PDD9am-L9og], [P],
s[INITIALIZING], reason [Failed to start shard, message
[IndexShardGatewayRecoveryException[[algotree][0] failed to recover shard];
nested: MapperParsingException[Failed to parse [Link]]; nested:
JsonParseException[Illegal unquoted character ((CTRL-CHAR, code 0)): has to
be escaped using backslash to be included in string value
at [Source: [B@11d565f; line: 1, column: 4062]]; ]]
[2012-06-22 17:36:14,922][WARN ][cluster.action.shard ] [Cat-Man]
received shard failed for [algotree][0], node[Rl9eKHxLRe6PDD9am-L9og], [P],
s[INITIALIZING], reason [Failed to start shard, message
[IndexShardGatewayRecoveryException[[algotree][0] failed to recover shard];
nested: MapperParsingException[Failed to parse [Link]]; nested:
JsonParseException[Illegal unquoted character ((CTRL-CHAR, code 0)): has to
be escaped using backslash to be included in string value
at [Source: [B@11d565f; line: 1, column: 4062]]; ]]
[2012-06-22 17:36:17,165][WARN ][indices.cluster ] [Cat-Man]
[algotree][1] failed to start shard
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException:
[algotree][1] failed to recover shard
at
org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:201)
at
org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:177)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: org.elasticsearch.index.mapper.MapperParsingException: Failed to
parse [Categories.Types.Facets.MoneyEvent.entity]
at
org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(AbstractFieldMapper.java:320)
at
org.elasticsearch.index.mapper.object.ObjectMapper.serializeValue(ObjectMapper.java:580)
at
org.elasticsearch.index.mapper.object.ObjectMapper.parse(ObjectMapper.java:444)
at
org.elasticsearch.index.mapper.object.ObjectMapper.serializeObject(ObjectMapper.java:492)
at
org.elasticsearch.index.mapper.object.ObjectMapper.serializeArray(ObjectMapper.java:560)
at
org.elasticsearch.index.mapper.object.ObjectMapper.parse(ObjectMapper.java:436)
at
org.elasticsearch.index.mapper.object.ObjectMapper.serializeObject(ObjectMapper.java:492)
at
org.elasticsearch.index.mapper.object.ObjectMapper.serializeArray(ObjectMapper.java:560)
at
org.elasticsearch.index.mapper.object.ObjectMapper.parse(ObjectMapper.java:436)
at
org.elasticsearch.index.mapper.object.ObjectMapper.serializeObject(ObjectMapper.java:492)
at
org.elasticsearch.index.mapper.object.ObjectMapper.parse(ObjectMapper.java:434)
at
org.elasticsearch.index.mapper.object.ObjectMapper.serializeObject(ObjectMapper.java:492)
at
org.elasticsearch.index.mapper.object.ObjectMapper.parse(ObjectMapper.java:434)
at
org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:473)
at
org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:417)
at
org.elasticsearch.index.shard.service.InternalIndexShard.prepareIndex(InternalIndexShard.java:311)
at
org.elasticsearch.index.shard.service.InternalIndexShard.performRecoveryOperation(InternalIndexShard.java:624)
at
org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:196)
... 4 more
.
What could have posssibly happened? How can we resolve this issue.. we
are using Ubuntu..is there a way that i can get the data back..

Thanks in advance......

nel

Pierrick · June 28, 2012, 7:57am

Hi,

Same problem for me.
Anyone has a solution ?

On Friday, June 22, 2012 3:36:16 PM UTC+2, nel wrote:

Hi

We are facing an isssue with ES. Our ES instance crashed due to low disk
space. After that it triggers out exceptions which are described below..

at
org.elasticsearch.index.shard.service.InternalIndexShard.performRecoveryOperation(InternalIndexShard.java:624)
at
org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:196)
... 4 more
Caused by: org.elasticsearch.common.jackson.JsonParseException: Illegal
unquoted character ((CTRL-CHAR, code 0)): has to be escaped using backslash
to be included in string value
at [Source: [B@bfc25c; line: 1, column: 4062]
at
org.elasticsearch.common.jackson.JsonParser._constructError(JsonParser.java:1432)
at
org.elasticsearch.common.jackson.impl.JsonParserMinimalBase._reportError(JsonParserMinimalBase.java:521)
at
org.elasticsearch.common.jackson.impl.JsonParserMinimalBase._throwUnquotedSpace(JsonParserMinimalBase.java:482)
at
org.elasticsearch.common.jackson.impl.Utf8StreamParser._finishString2(Utf8StreamParser.java:1975)
at
org.elasticsearch.common.jackson.impl.Utf8StreamParser._finishString(Utf8StreamParser.java:1905)
at
org.elasticsearch.common.jackson.impl.Utf8StreamParser.getText(Utf8StreamParser.java:276)
at
org.elasticsearch.common.xcontent.json.JsonXContentParser.text(JsonXContentParser.java:83)
at
org.elasticsearch.common.xcontent.support.AbstractXContentParser.textOrNull(AbstractXContentParser.java:106)
at
org.elasticsearch.index.mapper.core.StringFieldMapper.parseCreateField(StringFieldMapper.java:207)
at
org.elasticsearch.index.mapper.core.StringFieldMapper.parseCreateField(StringFieldMapper.java:42)
at
org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(AbstractFieldMapper.java:307)
... 11 more
[2012-06-22 17:36:11,075][WARN ][cluster.action.shard ] [Cat-Man]
sending failed shard for [algotree][0], node[Rl9eKHxLRe6PDD9am-L9og], [P],
s[INITIALIZING], reason [Failed to start shard, message
[IndexShardGatewayRecoveryException[[algotree][0] failed to recover shard];
nested: MapperParsingException[Failed to parse [Link]]; nested:
JsonParseException[Illegal unquoted character ((CTRL-CHAR, code 0)): has to
be escaped using backslash to be included in string value
at [Source: [B@bfc25c; line: 1, column: 4062]]; ]]
[2012-06-22 17:36:11,075][WARN ][cluster.action.shard ] [Cat-Man]
received shard failed for [algotree][0], node[Rl9eKHxLRe6PDD9am-L9og], [P],
s[INITIALIZING], reason [Failed to start shard, message
[IndexShardGatewayRecoveryException[[algotree][0] failed to recover shard];
nested: MapperParsingException[Failed to parse [Link]]; nested:
JsonParseException[Illegal unquoted character ((CTRL-CHAR, code 0)): has to
be escaped using backslash to be included in string value
at [Source: [B@bfc25c; line: 1, column: 4062]]; ]]
[2012-06-22 17:36:13,107][WARN ][indices.cluster ] [Cat-Man]
[algotree][1] failed to start shard
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException:
[algotree][1] failed to recover shard
at
org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:201)
at
org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:177)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: org.elasticsearch.index.mapper.MapperParsingException: Failed
to parse [Categories.Types.Facets.MoneyEvent.entity]
at
org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(AbstractFieldMapper.java:320)
at
org.elasticsearch.index.mapper.object.ObjectMapper.serializeValue(ObjectMapper.java:580)
at
org.elasticsearch.index.mapper.object.ObjectMapper.parse(ObjectMapper.java:444)
at
org.elasticsearch.index.mapper.object.ObjectMapper.serializeObject(ObjectMapper.java:492)
at
org.elasticsearch.index.mapper.object.ObjectMapper.serializeArray(ObjectMapper.java:560)
at
org.elasticsearch.index.mapper.object.ObjectMapper.parse(ObjectMapper.java:436)
at
org.elasticsearch.index.mapper.object.ObjectMapper.serializeObject(ObjectMapper.java:492)
at
org.elasticsearch.index.mapper.object.ObjectMapper.serializeArray(ObjectMapper.java:560)
at
org.elasticsearch.index.mapper.object.ObjectMapper.parse(ObjectMapper.java:436)
at
org.elasticsearch.index.mapper.object.ObjectMapper.serializeObject(ObjectMapper.java:492)
at
org.elasticsearch.index.mapper.object.ObjectMapper.parse(ObjectMapper.java:434)
at
org.elasticsearch.index.mapper.object.ObjectMapper.serializeObject(ObjectMapper.java:492)
at
org.elasticsearch.index.mapper.object.ObjectMapper.parse(ObjectMapper.java:434)
at
org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:473)
at
org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:417)
at
org.elasticsearch.index.shard.service.InternalIndexShard.prepareIndex(InternalIndexShard.java:311)
at
org.elasticsearch.index.shard.service.InternalIndexShard.performRecoveryOperation(InternalIndexShard.java:624)
at
org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:196)
... 4 more
Caused by: org.elasticsearch.common.jackson.JsonParseException: Illegal
unquoted character ((CTRL-CHAR, code 0)): has to be escaped using backslash
to be included in string value
at [Source: [B@eb203b; line: 1, column: 937]
at
org.elasticsearch.common.jackson.JsonParser._constructError(JsonParser.java:1432)
at
org.elasticsearch.common.jackson.impl.JsonParserMinimalBase._reportError(JsonParserMinimalBase.java:521)
at
org.elasticsearch.common.jackson.impl.JsonParserMinimalBase._throwUnquotedSpace(JsonParserMinimalBase.java:482)
at
org.elasticsearch.common.jackson.impl.Utf8StreamParser._finishString2(Utf8StreamParser.java:1975)
at
org.elasticsearch.common.jackson.impl.Utf8StreamParser._finishString(Utf8StreamParser.java:1905)
at
org.elasticsearch.common.jackson.impl.Utf8StreamParser.getText(Utf8StreamParser.java:276)
at
org.elasticsearch.common.xcontent.json.JsonXContentParser.text(JsonXContentParser.java:83)
at
org.elasticsearch.common.xcontent.support.AbstractXContentParser.textOrNull(AbstractXContentParser.java:106)
at
org.elasticsearch.index.mapper.core.StringFieldMapper.parseCreateField(StringFieldMapper.java:207)
at
org.elasticsearch.index.mapper.core.StringFieldMapper.parseCreateField(StringFieldMapper.java:42)
at
org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(AbstractFieldMapper.java:307)
... 21 more
[2012-06-22 17:36:13,135][WARN ][cluster.action.shard ] [Cat-Man]
sending failed shard for [algotree][1], node[Rl9eKHxLRe6PDD9am-L9og], [P],
s[INITIALIZING], reason [Failed to start shard, message
[IndexShardGatewayRecoveryException[[algotree][1] failed to recover shard];
nested: MapperParsingException[Failed to parse
[Categories.Types.Facets.MoneyEvent.entity]]; nested:
JsonParseException[Illegal unquoted character ((CTRL-CHAR, code 0)): has to
be escaped using backslash to be included in string value
at [Source: [B@eb203b; line: 1, column: 937]]; ]]
[2012-06-22 17:36:13,135][WARN ][cluster.action.shard ] [Cat-Man]
received shard failed for [algotree][1], node[Rl9eKHxLRe6PDD9am-L9og], [P],
s[INITIALIZING], reason [Failed to start shard, message
[IndexShardGatewayRecoveryException[[algotree][1] failed to recover shard];
nested: MapperParsingException[Failed to parse
[Categories.Types.Facets.MoneyEvent.entity]]; nested:
JsonParseException[Illegal unquoted character ((CTRL-CHAR, code 0)): has to
be escaped using backslash to be included in string value
at [Source: [B@eb203b; line: 1, column: 937]]; ]]
[2012-06-22 17:36:14,895][WARN ][indices.cluster ] [Cat-Man]
[algotree][0] failed to start shard
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException:
[algotree][0] failed to recover shard
at
org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:201)
at
org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:177)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: org.elasticsearch.index.mapper.MapperParsingException: Failed
to parse [Link]
at
org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(AbstractFieldMapper.java:320)
at
org.elasticsearch.index.mapper.object.ObjectMapper.serializeValue(ObjectMapper.java:580)
at
org.elasticsearch.index.mapper.object.ObjectMapper.parse(ObjectMapper.java:444)
at
org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:473)
at
org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:417)
at
org.elasticsearch.index.shard.service.InternalIndexShard.prepareIndex(InternalIndexShard.java:311)
at
org.elasticsearch.index.shard.service.InternalIndexShard.performRecoveryOperation(InternalIndexShard.java:624)
at
org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:196)
... 4 more
Caused by: org.elasticsearch.common.jackson.JsonParseException: Illegal
unquoted character ((CTRL-CHAR, code 0)): has to be escaped using backslash
to be included in string value
at [Source: [B@11d565f; line: 1, column: 4062]
at
org.elasticsearch.common.jackson.JsonParser._constructError(JsonParser.java:1432)
at
org.elasticsearch.common.jackson.impl.JsonParserMinimalBase._reportError(JsonParserMinimalBase.java:521)
at
org.elasticsearch.common.jackson.impl.JsonParserMinimalBase._throwUnquotedSpace(JsonParserMinimalBase.java:482)
at
org.elasticsearch.common.jackson.impl.Utf8StreamParser._finishString2(Utf8StreamParser.java:1975)
at
org.elasticsearch.common.jackson.impl.Utf8StreamParser._finishString(Utf8StreamParser.java:1905)
at
org.elasticsearch.common.jackson.impl.Utf8StreamParser.getText(Utf8StreamParser.java:276)
at
org.elasticsearch.common.xcontent.json.JsonXContentParser.text(JsonXContentParser.java:83)
at
org.elasticsearch.common.xcontent.support.AbstractXContentParser.textOrNull(AbstractXContentParser.java:106)
at
org.elasticsearch.index.mapper.core.StringFieldMapper.parseCreateField(StringFieldMapper.java:207)
at
org.elasticsearch.index.mapper.core.StringFieldMapper.parseCreateField(StringFieldMapper.java:42)
at
org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(AbstractFieldMapper.java:307)
... 11 more
[2012-06-22 17:36:14,922][WARN ][cluster.action.shard ] [Cat-Man]
sending failed shard for [algotree][0], node[Rl9eKHxLRe6PDD9am-L9og], [P],
s[INITIALIZING], reason [Failed to start shard, message
[IndexShardGatewayRecoveryException[[algotree][0] failed to recover shard];
nested: MapperParsingException[Failed to parse [Link]]; nested:
JsonParseException[Illegal unquoted character ((CTRL-CHAR, code 0)): has to
be escaped using backslash to be included in string value
at [Source: [B@11d565f; line: 1, column: 4062]]; ]]
[2012-06-22 17:36:14,922][WARN ][cluster.action.shard ] [Cat-Man]
received shard failed for [algotree][0], node[Rl9eKHxLRe6PDD9am-L9og], [P],
s[INITIALIZING], reason [Failed to start shard, message
[IndexShardGatewayRecoveryException[[algotree][0] failed to recover shard];
nested: MapperParsingException[Failed to parse [Link]]; nested:
JsonParseException[Illegal unquoted character ((CTRL-CHAR, code 0)): has to
be escaped using backslash to be included in string value
at [Source: [B@11d565f; line: 1, column: 4062]]; ]]
[2012-06-22 17:36:17,165][WARN ][indices.cluster ] [Cat-Man]
[algotree][1] failed to start shard
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException:
[algotree][1] failed to recover shard
at
org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:201)
at
org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:177)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: org.elasticsearch.index.mapper.MapperParsingException: Failed
to parse [Categories.Types.Facets.MoneyEvent.entity]
at
org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(AbstractFieldMapper.java:320)
at
org.elasticsearch.index.mapper.object.ObjectMapper.serializeValue(ObjectMapper.java:580)
at
org.elasticsearch.index.mapper.object.ObjectMapper.parse(ObjectMapper.java:444)
at
org.elasticsearch.index.mapper.object.ObjectMapper.serializeObject(ObjectMapper.java:492)
at
org.elasticsearch.index.mapper.object.ObjectMapper.serializeArray(ObjectMapper.java:560)
at
org.elasticsearch.index.mapper.object.ObjectMapper.parse(ObjectMapper.java:436)
at
org.elasticsearch.index.mapper.object.ObjectMapper.serializeObject(ObjectMapper.java:492)
at
org.elasticsearch.index.mapper.object.ObjectMapper.serializeArray(ObjectMapper.java:560)
at
org.elasticsearch.index.mapper.object.ObjectMapper.parse(ObjectMapper.java:436)
at
org.elasticsearch.index.mapper.object.ObjectMapper.serializeObject(ObjectMapper.java:492)
at
org.elasticsearch.index.mapper.object.ObjectMapper.parse(ObjectMapper.java:434)
at
org.elasticsearch.index.mapper.object.ObjectMapper.serializeObject(ObjectMapper.java:492)
at
org.elasticsearch.index.mapper.object.ObjectMapper.parse(ObjectMapper.java:434)
at
org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:473)
at
org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:417)
at
org.elasticsearch.index.shard.service.InternalIndexShard.prepareIndex(InternalIndexShard.java:311)
at
org.elasticsearch.index.shard.service.InternalIndexShard.performRecoveryOperation(InternalIndexShard.java:624)
at
org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:196)
... 4 more
.
What could have posssibly happened? How can we resolve this issue.. we
are using Ubuntu..is there a way that i can get the data back..

Thanks in advance......

nel

jagdeep · June 28, 2012, 10:17am

Please specify ES setup, number of nodes, shards etc.
Is it not able to recover after hard restart?

On Jun 22, 6:36 pm, nel NeilA...@algotree.com wrote:

Hi

We are facing an isssue with ES. Our ES instance crashed due to low disk
space. After that it triggers out exceptions which are described below..

at
org.elasticsearch.index.shard.service.InternalIndexShard.performRecoveryOpe ration(InternalIndexShard.java:624)
at
org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalI ndexShardGateway.java:196)
... 4 more
Caused by: org.elasticsearch.common.jackson.JsonParseException: Illegal
unquoted character ((CTRL-CHAR, code 0)): has to be escaped using backslash
to be included in string value
at [Source: [B@bfc25c; line: 1, column: 4062]
at
org.elasticsearch.common.jackson.JsonParser._constructError(JsonParser.java :1432)
at
org.elasticsearch.common.jackson.impl.JsonParserMinimalBase._reportError(Js onParserMinimalBase.java:521)
at
org.elasticsearch.common.jackson.impl.JsonParserMinimalBase._throwUnquotedS pace(JsonParserMinimalBase.java:482)
at
org.elasticsearch.common.jackson.impl.Utf8StreamParser._finishString2(Utf8S treamParser.java:1975)
at
org.elasticsearch.common.jackson.impl.Utf8StreamParser._finishString(Utf8St reamParser.java:1905)
at
org.elasticsearch.common.jackson.impl.Utf8StreamParser.getText(Utf8StreamPa rser.java:276)
at
org.elasticsearch.common.xcontent.json.JsonXContentParser.text(JsonXContent Parser.java:83)
at
org.elasticsearch.common.xcontent.support.AbstractXContentParser.textOrNull (AbstractXContentParser.java:106)
at
org.elasticsearch.index.mapper.core.StringFieldMapper.parseCreateField(Stri ngFieldMapper.java:207)
at
org.elasticsearch.index.mapper.core.StringFieldMapper.parseCreateField(Stri ngFieldMapper.java:42)
at
org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(AbstractField Mapper.java:307)
... 11 more
[2012-06-22 17:36:11,075][WARN ][cluster.action.shard ] [Cat-Man]
sending failed shard for [algotree][0], node[Rl9eKHxLRe6PDD9am-L9og], [P],
s[INITIALIZING], reason [Failed to start shard, message
[IndexShardGatewayRecoveryException[[algotree][0] failed to recover shard];
nested: MapperParsingException[Failed to parse [Link]]; nested:
JsonParseException[Illegal unquoted character ((CTRL-CHAR, code 0)): has to
be escaped using backslash to be included in string value
at [Source: [B@bfc25c; line: 1, column: 4062]]; ]]
[2012-06-22 17:36:11,075][WARN ][cluster.action.shard ] [Cat-Man]
received shard failed for [algotree][0], node[Rl9eKHxLRe6PDD9am-L9og], [P],
s[INITIALIZING], reason [Failed to start shard, message
[IndexShardGatewayRecoveryException[[algotree][0] failed to recover shard];
nested: MapperParsingException[Failed to parse [Link]]; nested:
JsonParseException[Illegal unquoted character ((CTRL-CHAR, code 0)): has to
be escaped using backslash to be included in string value
at [Source: [B@bfc25c; line: 1, column: 4062]]; ]]
[2012-06-22 17:36:13,107][WARN ][indices.cluster ] [Cat-Man]
[algotree][1] failed to start shard
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException:
[algotree][1] failed to recover shard
at
org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalI ndexShardGateway.java:201)
at
org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGa tewayService.java:177)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.j ava:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java: 908)
at java.lang.Thread.run(Thread.java:662)
Caused by: org.elasticsearch.index.mapper.MapperParsingException: Failed to
parse [Categories.Types.Facets.MoneyEvent.entity]
at
org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(AbstractField Mapper.java:320)
at
org.elasticsearch.index.mapper.object.ObjectMapper.serializeValue(ObjectMap per.java:580)
at
org.elasticsearch.index.mapper.object.ObjectMapper.parse(ObjectMapper.java: 444)
at
org.elasticsearch.index.mapper.object.ObjectMapper.serializeObject(ObjectMa pper.java:492)
at
org.elasticsearch.index.mapper.object.ObjectMapper.serializeArray(ObjectMap per.java:560)
at
org.elasticsearch.index.mapper.object.ObjectMapper.parse(ObjectMapper.java: 436)
at
org.elasticsearch.index.mapper.object.ObjectMapper.serializeObject(ObjectMa pper.java:492)
at
org.elasticsearch.index.mapper.object.ObjectMapper.serializeArray(ObjectMap per.java:560)
at
org.elasticsearch.index.mapper.object.ObjectMapper.parse(ObjectMapper.java: 436)
at
org.elasticsearch.index.mapper.object.ObjectMapper.serializeObject(ObjectMa pper.java:492)
at
org.elasticsearch.index.mapper.object.ObjectMapper.parse(ObjectMapper.java: 434)
at
org.elasticsearch.index.mapper.object.ObjectMapper.serializeObject(ObjectMa pper.java:492)
at
org.elasticsearch.index.mapper.object.ObjectMapper.parse(ObjectMapper.java: 434)
at
org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:473 )
at
org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:417 )
at
org.elasticsearch.index.shard.service.InternalIndexShard.prepareIndex(Inter nalIndexShard.java:311)
at
org.elasticsearch.index.shard.service.InternalIndexShard.performRecoveryOpe ration(InternalIndexShard.java:624)
at
org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalI ndexShardGateway.java:196)
... 4 more
Caused by: org.elasticsearch.common.jackson.JsonParseException: Illegal
unquoted character ((CTRL-CHAR, code 0)): has to be escaped using backslash
to be included in string value
at [Source: [B@eb203b; line: 1, column: 937]
at
org.elasticsearch.common.jackson.JsonParser._constructError(JsonParser.java :1432)
at
org.elasticsearch.common.jackson.impl.JsonParserMinimalBase._reportError(Js onParserMinimalBase.java:521)
at
org.elasticsearch.common.jackson.impl.JsonParserMinimalBase._throwUnquotedS pace(JsonParserMinimalBase.java:482)
at
org.elasticsearch.common.jackson.impl.Utf8StreamParser._finishString2(Utf8S treamParser.java:1975)
at
org.elasticsearch.common.jackson.impl.Utf8StreamParser._finishString(Utf8St reamParser.java:1905)
at
org.elasticsearch.common.jackson.impl.Utf8StreamParser.getText(Utf8StreamPa rser.java:276)
at
org.elasticsearch.common.xcontent.json.JsonXContentParser.text(JsonXContent Parser.java:83)
at
org.elasticsearch.common.xcontent.support.AbstractXContentParser.textOrNull (AbstractXContentParser.java:106)
at
org.elasticsearch.index.mapper.core.StringFieldMapper.parseCreateField(Stri ngFieldMapper.java:207)
at
org.elasticsearch.index.mapper.core.StringFieldMapper.parseCreateField(Stri ngFieldMapper.java:42)
at
org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(AbstractField Mapper.java:307)
... 21 more
[2012-06-22 17:36:13,135][WARN ][cluster.action.shard ] [Cat-Man]
sending failed shard for [algotree][1], node[Rl9eKHxLRe6PDD9am-L9og], [P],
s[INITIALIZING], reason [Failed to start shard, message
[IndexShardGatewayRecoveryException[[algotree][1] failed to recover shard];
nested: MapperParsingException[Failed to parse
[Categories.Types.Facets.MoneyEvent.entity]]; nested:
JsonParseException[Illegal unquoted character ((CTRL-CHAR, code 0)): has to
be escaped using backslash to be included in string value
at [Source: [B@eb203b; line: 1, column: 937]]; ]]
[2012-06-22 17:36:13,135][WARN ][cluster.action.shard ] [Cat-Man]
received shard failed for [algotree][1], node[Rl9eKHxLRe6PDD9am-L9og], [P],
s[INITIALIZING], reason [Failed to start shard, message
[IndexShardGatewayRecoveryException[[algotree][1] failed to recover shard];
nested: MapperParsingException[Failed to parse
[Categories.Types.Facets.MoneyEvent.entity]]; nested:
JsonParseException[Illegal unquoted character ((CTRL-CHAR, code 0)): has to
be escaped using backslash to be included in string value
at [Source: [B@eb203b; line: 1, column: 937]]; ]]
[2012-06-22 17:36:14,895][WARN ][indices.cluster ] [Cat-Man]
[algotree][0] failed to start shard
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException:
[algotree][0] failed to recover shard
at
org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalI ndexShardGateway.java:201)
at
org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGa tewayService.java:177)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.j ava:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java: 908)
at java.lang.Thread.run(Thread.java:662)
Caused by: org.elasticsearch.index.mapper.MapperParsingException: Failed to
parse [Link]
at
org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(AbstractField Mapper.java:320)
at
org.elasticsearch.index.mapper.object.ObjectMapper.serializeValue(ObjectMap per.java:580)
at
org.elasticsearch.index.mapper.object.ObjectMapper.parse(ObjectMapper.java: 444)
at
org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:473 )
at
org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:417 )
at
org.elasticsearch.index.shard.service.InternalIndexShard.prepareIndex(Inter nalIndexShard.java:311)
at
org.elasticsearch.index.shard.service.InternalIndexShard.performRecoveryOpe ration(InternalIndexShard.java:624)
at
org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalI ndexShardGateway.java:196)
... 4 more
Caused by: org.elasticsearch.common.jackson.JsonParseException: Illegal
unquoted character ((CTRL-CHAR, code 0)): has to be escaped using backslash
to be included in string value
at [Source: [B@11d565f; line: 1, column: 4062]
at
org.elasticsearch.common.jackson.JsonParser._constructError(JsonParser.java :1432)
at
org.elasticsearch.common.jackson.impl.JsonParserMinimalBase._reportError(Js onParserMinimalBase.java:521)
at
...

read more »

Clinton_Gormley · June 28, 2012, 10:40am

Hiya

We are facing an isssue with ES. Our ES instance crashed due to low
disk space. After that it triggers out exceptions which are described
below..

shard]; nested: MapperParsingException[Failed to parse [Link]];
nested: JsonParseException[Illegal unquoted character ((CTRL-CHAR,
code 0)): has to be escaped using backslash to be included in string
value

I'm guessing here, but it looks like your mapping has been corrupted,
presumably by running out of disk space.

You may be able to fix it by looking in
data/nodes/0/indices/INDEXNAME/_state. But that file is compressed, I'm
not sure what format. Do you have an older copy that you might be able
to use?

And it sounds a lot like you were only running one instance, so no other
instance you could copy the mapping from

clint

Igor_Motov · June 28, 2012, 3:16pm

In nel's case it was corrupted transaction log. When you run out of disk
space sometimes the last transaction cannot be fully written into
transaction log and then it fails on recovery. If you see exactly the same
error messages, you can try the following:

shut down elasticsearch cluster
find all shards that cannot recover by searching log file
for each shard move its non-zero length translog file into a temporary
directory (see explanation below)
start elasticsearch cluster
if you see messages for other shards - repeat

If you see message like this:

[2012-06-22 17:36:17,165][WARN ][indices.cluster ] [Cat-Man] [*
myindex*][1] failed to start shard

It means that it cannot recover shard 1 of the index myindex on the
node Cat-Man. If you take a look at data/elasticsearch/nodes/0/indices/*
myindex/1*/translog directory, you will find files like this:
translog-123456677899 or translog-123456677899.recovering. One of them will
have non-zero length. Move it to a temporary directory and try starting the
server.

The transaction log files that you will be moving out contain your most
recently updated and indexed documents. So, these updates will be lost as a
result of this operations, but you should be able to recover the rest of
your data.

On Thursday, June 28, 2012 6:40:06 AM UTC-4, Clinton Gormley wrote:

Hiya

We are facing an isssue with ES. Our ES instance crashed due to low
disk space. After that it triggers out exceptions which are described
below..

shard]; nested: MapperParsingException[Failed to parse [Link]];
nested: JsonParseException[Illegal unquoted character ((CTRL-CHAR,
code 0)): has to be escaped using backslash to be included in string
value

I'm guessing here, but it looks like your mapping has been corrupted,
presumably by running out of disk space.

You may be able to fix it by looking in
data/nodes/0/indices/INDEXNAME/_state. But that file is compressed, I'm
not sure what format. Do you have an older copy that you might be able
to use?

And it sounds a lot like you were only running one instance, so no other
instance you could copy the mapping from

clint

Laurent_Block · July 4, 2012, 9:35am

Hi there,

We have a similar problem with our graylog2 constellation.
We have 2 nodes in master-master, during the weekend (regarding a huge
processing of files) we also ran out of space on the harddisks. We enlarged
the partitions in order to get again some free space on the disks.
Since then, we have a CRITICAL error on our monitoring system which says
that string "green" cannot be found on the health status report of the
cluster, here's the curl output from the server:

{
"cluster_name" : "graylog2",
"status" : "red",
"timed_out" : false,
"number_of_nodes" : 2,
"number_of_data_nodes" : 2,
"active_primary_shards" : 1,
"active_shards" : 2,
"relocating_shards" : 0,
"initializing_shards" : 1,
"unassigned_shards" : 1
}

As we have 1 unassigned shard, I run deeper in debugging and found this in
the logs:

Server1:

[WARN ][cluster.action.shard ] [logserver01] received shard failed for
[graylog2][0], node[B3AFfzhhQ8yMJmuaJF6v6g], [P], s[INITIALIZING], reason
[Failed to start shard, message
[IndexShardGatewayRecoveryException[[graylog2][0] failed to recover shard];
nested: MapperParsingException[Failed to parse [full_message]]; nested:
JsonParseException[Illegal unquoted character ((CTRL-CHAR, code 0)): has to
be escaped using backslash to be included in string value at [Source:
[B@4bd8d548; line: 1, column: 575]]; ]]

Server2:

[WARN ][cluster.action.shard ] [logserver02] sending failed shard for
[graylog2][0], node[lNCk9JoQTIKrT76k2B1pUw], [P], s[INITIALIZING], reason
[Failed to start shard, message
[IndexShardGatewayRecoveryException[[graylog2][0] failed to recover shard];
nested: MapperParsingException[Failed to parse [message]]; nested:
JsonParseException[Illegal unquoted character ((CTRL-CHAR, code 0)): has to
be escaped using backslash to be included in string value at [Source:
[B@77a202bf; line: 1, column: 79]]; ]]

Regarding the last comment that Igor wrote, do I only have to do the
procedure he described on the second server where the "sending" failed, or
on the whole cluster?

I'm really a newbie on elasticsearch so sorry if I cannot provide you more
professional input.

On Thursday, June 28, 2012 5:16:23 PM UTC+2, Igor Motov wrote:

In nel's case it was corrupted transaction log. When you run out of disk
space sometimes the last transaction cannot be fully written into
transaction log and then it fails on recovery. If you see exactly the same
error messages, you can try the following:

shut down elasticsearch cluster

find all shards that cannot recover by searching log file

for each shard move its non-zero length translog file into a temporary
directory (see explanation below)

start elasticsearch cluster

if you see messages for other shards - repeat

If you see message like this:

[2012-06-22 17:36:17,165][WARN ][indices.cluster ] [Cat-Man] [*
myindex*][1] failed to start shard

It means that it cannot recover shard 1 of the index myindex on the
node Cat-Man. If you take a look at data/elasticsearch/nodes/0/indices/*
myindex/1*/translog directory, you will find files like this:
translog-123456677899 or translog-123456677899.recovering. One of them will
have non-zero length. Move it to a temporary directory and try starting the
server.

The transaction log files that you will be moving out contain your most
recently updated and indexed documents. So, these updates will be lost as a
result of this operations, but you should be able to recover the rest of
your data.

On Thursday, June 28, 2012 6:40:06 AM UTC-4, Clinton Gormley wrote:

Hiya

We are facing an isssue with ES. Our ES instance crashed due to low
disk space. After that it triggers out exceptions which are described
below..

shard]; nested: MapperParsingException[Failed to parse [Link]];
nested: JsonParseException[Illegal unquoted character ((CTRL-CHAR,
code 0)): has to be escaped using backslash to be included in string
value

I'm guessing here, but it looks like your mapping has been corrupted,
presumably by running out of disk space.

You may be able to fix it by looking in
data/nodes/0/indices/INDEXNAME/_state. But that file is compressed, I'm
not sure what format. Do you have an older copy that you might be able
to use?

And it sounds a lot like you were only running one instance, so no other
instance you could copy the mapping from

clint

Laurent_Block · July 4, 2012, 1:06pm

A more explicit output from the cluster health here:

{
"cluster_name" : "graylog2",
"status" : "red",
"timed_out" : false,
"number_of_nodes" : 2,
"number_of_data_nodes" : 2,
"active_primary_shards" : 1,
"active_shards" : 2,
"relocating_shards" : 0,
"initializing_shards" : 1,
"unassigned_shards" : 1,
"indices" : {
"graylog2" : {
"status" : "red",
"number_of_shards" : 2,
"number_of_replicas" : 1,
"active_primary_shards" : 1,
"active_shards" : 2,
"relocating_shards" : 0,
"initializing_shards" : 1,
"unassigned_shards" : 1,
"shards" : {
"0" : {
"status" : "red",
"primary_active" : false,
"active_shards" : 0,
"relocating_shards" : 0,
"initializing_shards" : 1,
"unassigned_shards" : 1
},
"1" : {
"status" : "green",
"primary_active" : true,
"active_shards" : 2,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0
}
}
}
}
}

Now at least I know there the problem is, on shard 0 of my graylog2 cluster.
Here's an extract of the file system (translog folder):

Server1:
-rw-rw-r-- 1 elasticsearch elasticsearch 323580 Jul 4 15:05
translog-1337094255321
-rw-rw-r-- 1 elasticsearch elasticsearch 1142760 Jun 29 18:33
translog-1337094255321.recovering

Server2:
total 516
-rw-rw-r-- 1 elasticsearch elasticsearch 0 Jul 4 15:05
translog-1337094255544
-rw-rw-r-- 1 elasticsearch elasticsearch 524288 Jul 2 10:08
translog-1337094255544.recovering

Any advice what I can do now?
Many thanks for your help in advance.

phoenix

On Wednesday, July 4, 2012 11:35:39 AM UTC+2, phoenix2lux wrote:

Hi there,

We have a similar problem with our graylog2 constellation.
We have 2 nodes in master-master, during the weekend (regarding a huge
processing of files) we also ran out of space on the harddisks. We enlarged
the partitions in order to get again some free space on the disks.
Since then, we have a CRITICAL error on our monitoring system which says
that string "green" cannot be found on the health status report of the
cluster, here's the curl output from the server:

{
"cluster_name" : "graylog2",
"status" : "red",
"timed_out" : false,
"number_of_nodes" : 2,
"number_of_data_nodes" : 2,
"active_primary_shards" : 1,
"active_shards" : 2,
"relocating_shards" : 0,
"initializing_shards" : 1,
"unassigned_shards" : 1
}

As we have 1 unassigned shard, I run deeper in debugging and found this in
the logs:

Server1:

[WARN ][cluster.action.shard ] [logserver01] received shard failed for
[graylog2][0], node[B3AFfzhhQ8yMJmuaJF6v6g], [P], s[INITIALIZING], reason
[Failed to start shard, message
[IndexShardGatewayRecoveryException[[graylog2][0] failed to recover shard];
nested: MapperParsingException[Failed to parse [full_message]]; nested:
JsonParseException[Illegal unquoted character ((CTRL-CHAR, code 0)): has to
be escaped using backslash to be included in string value at [Source:
[B@4bd8d548; line: 1, column: 575]]; ]]

Server2:

[WARN ][cluster.action.shard ] [logserver02] sending failed shard for
[graylog2][0], node[lNCk9JoQTIKrT76k2B1pUw], [P], s[INITIALIZING], reason
[Failed to start shard, message
[IndexShardGatewayRecoveryException[[graylog2][0] failed to recover shard];
nested: MapperParsingException[Failed to parse [message]]; nested:
JsonParseException[Illegal unquoted character ((CTRL-CHAR, code 0)): has to
be escaped using backslash to be included in string value at [Source:
[B@77a202bf; line: 1, column: 79]]; ]]

Regarding the last comment that Igor wrote, do I only have to do the
procedure he described on the second server where the "sending" failed, or
on the whole cluster?

I'm really a newbie on elasticsearch so sorry if I cannot provide you more
professional input.

Igor_Motov · July 8, 2012, 3:19pm

Hi phoenix,

First of all I would suggest to stop all indexing until cluster is
recovered. Then I would start with removing all translog-.recovering files
on all shards that report issues in the log and if this wouldn't help -
removing all non-zero length translog- files on these shards.

Igor

On Wednesday, July 4, 2012 9:06:31 AM UTC-4, phoenix2lux wrote:

A more explicit output from the cluster health here:

{
"cluster_name" : "graylog2",
"status" : "red",
"timed_out" : false,
"number_of_nodes" : 2,
"number_of_data_nodes" : 2,
"active_primary_shards" : 1,
"active_shards" : 2,
"relocating_shards" : 0,
"initializing_shards" : 1,
"unassigned_shards" : 1,
"indices" : {
"graylog2" : {
"status" : "red",
"number_of_shards" : 2,
"number_of_replicas" : 1,
"active_primary_shards" : 1,
"active_shards" : 2,
"relocating_shards" : 0,
"initializing_shards" : 1,
"unassigned_shards" : 1,
"shards" : {
"0" : {
"status" : "red",
"primary_active" : false,
"active_shards" : 0,
"relocating_shards" : 0,
"initializing_shards" : 1,
"unassigned_shards" : 1
},
"1" : {
"status" : "green",
"primary_active" : true,
"active_shards" : 2,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0
}
}
}
}
}

Now at least I know there the problem is, on shard 0 of my graylog2
cluster.
Here's an extract of the file system (translog folder):

Server1:
-rw-rw-r-- 1 elasticsearch elasticsearch 323580 Jul 4 15:05
translog-1337094255321
-rw-rw-r-- 1 elasticsearch elasticsearch 1142760 Jun 29 18:33
translog-1337094255321.recovering

Server2:
total 516
-rw-rw-r-- 1 elasticsearch elasticsearch 0 Jul 4 15:05
translog-1337094255544
-rw-rw-r-- 1 elasticsearch elasticsearch 524288 Jul 2 10:08
translog-1337094255544.recovering

Any advice what I can do now?
Many thanks for your help in advance.

phoenix

On Wednesday, July 4, 2012 11:35:39 AM UTC+2, phoenix2lux wrote:

Hi there,

We have a similar problem with our graylog2 constellation.
We have 2 nodes in master-master, during the weekend (regarding a huge
processing of files) we also ran out of space on the harddisks. We enlarged
the partitions in order to get again some free space on the disks.
Since then, we have a CRITICAL error on our monitoring system which says
that string "green" cannot be found on the health status report of the
cluster, here's the curl output from the server:

{
"cluster_name" : "graylog2",
"status" : "red",
"timed_out" : false,
"number_of_nodes" : 2,
"number_of_data_nodes" : 2,
"active_primary_shards" : 1,
"active_shards" : 2,
"relocating_shards" : 0,
"initializing_shards" : 1,
"unassigned_shards" : 1
}

As we have 1 unassigned shard, I run deeper in debugging and found this
in the logs:

Server1:

[WARN ][cluster.action.shard ] [logserver01] received shard failed
for [graylog2][0], node[B3AFfzhhQ8yMJmuaJF6v6g], [P], s[INITIALIZING],
reason [Failed to start shard, message
[IndexShardGatewayRecoveryException[[graylog2][0] failed to recover shard];
nested: MapperParsingException[Failed to parse [full_message]]; nested:
JsonParseException[Illegal unquoted character ((CTRL-CHAR, code 0)): has to
be escaped using backslash to be included in string value at [Source:
[B@4bd8d548; line: 1, column: 575]]; ]]

Server2:

[WARN ][cluster.action.shard ] [logserver02] sending failed shard for
[graylog2][0], node[lNCk9JoQTIKrT76k2B1pUw], [P], s[INITIALIZING], reason
[Failed to start shard, message
[IndexShardGatewayRecoveryException[[graylog2][0] failed to recover shard];
nested: MapperParsingException[Failed to parse [message]]; nested:
JsonParseException[Illegal unquoted character ((CTRL-CHAR, code 0)): has to
be escaped using backslash to be included in string value at [Source:
[B@77a202bf; line: 1, column: 79]]; ]]

Regarding the last comment that Igor wrote, do I only have to do the
procedure he described on the second server where the "sending" failed, or
on the whole cluster?

I'm really a newbie on elasticsearch so sorry if I cannot provide you
more professional input.

Laurent_Block · July 8, 2012, 3:50pm

Hi Igor,
I followed all these steps and after waiting couple of minutes for the re-indexing of the ES cluster, the health of the cluster went green again and all shards are active and synced.

Many thanks for your help!

On 08.07.2012, at 17:19, Igor Motov imotov@gmail.com wrote:

Hi phoenix,

First of all I would suggest to stop all indexing until cluster is recovered. Then I would start with removing all translog-.recovering files on all shards that report issues in the log and if this wouldn't help - removing all non-zero length translog- files on these shards.

Igor

On Wednesday, July 4, 2012 9:06:31 AM UTC-4, phoenix2lux wrote:

A more explicit output from the cluster health here:

{
"cluster_name" : "graylog2",
"status" : "red",
"timed_out" : false,
"number_of_nodes" : 2,
"number_of_data_nodes" : 2,
"active_primary_shards" : 1,
"active_shards" : 2,
"relocating_shards" : 0,
"initializing_shards" : 1,
"unassigned_shards" : 1,
"indices" : {
"graylog2" : {
"status" : "red",
"number_of_shards" : 2,
"number_of_replicas" : 1,
"active_primary_shards" : 1,
"active_shards" : 2,
"relocating_shards" : 0,
"initializing_shards" : 1,
"unassigned_shards" : 1,
"shards" : {
"0" : {
"status" : "red",
"primary_active" : false,
"active_shards" : 0,
"relocating_shards" : 0,
"initializing_shards" : 1,
"unassigned_shards" : 1
},
"1" : {
"status" : "green",
"primary_active" : true,
"active_shards" : 2,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0
}
}
}
}
}

Now at least I know there the problem is, on shard 0 of my graylog2 cluster.
Here's an extract of the file system (translog folder):

Server1:
-rw-rw-r-- 1 elasticsearch elasticsearch 323580 Jul 4 15:05 translog-1337094255321
-rw-rw-r-- 1 elasticsearch elasticsearch 1142760 Jun 29 18:33 translog-1337094255321.recovering

Server2:
total 516
-rw-rw-r-- 1 elasticsearch elasticsearch 0 Jul 4 15:05 translog-1337094255544
-rw-rw-r-- 1 elasticsearch elasticsearch 524288 Jul 2 10:08 translog-1337094255544.recovering

Any advice what I can do now?
Many thanks for your help in advance.

phoenix

On Wednesday, July 4, 2012 11:35:39 AM UTC+2, phoenix2lux wrote:

Hi there,

We have a similar problem with our graylog2 constellation.
We have 2 nodes in master-master, during the weekend (regarding a huge processing of files) we also ran out of space on the harddisks. We enlarged the partitions in order to get again some free space on the disks.
Since then, we have a CRITICAL error on our monitoring system which says that string "green" cannot be found on the health status report of the cluster, here's the curl output from the server:

{
"cluster_name" : "graylog2",
"status" : "red",
"timed_out" : false,
"number_of_nodes" : 2,
"number_of_data_nodes" : 2,
"active_primary_shards" : 1,
"active_shards" : 2,
"relocating_shards" : 0,
"initializing_shards" : 1,
"unassigned_shards" : 1
}

As we have 1 unassigned shard, I run deeper in debugging and found this in the logs:

Server1:
[WARN ][cluster.action.shard ] [logserver01] received shard failed for [graylog2][0], node[B3AFfzhhQ8yMJmuaJF6v6g], [P], s[INITIALIZING], reason [Failed to start shard, message [IndexShardGatewayRecoveryException[[graylog2][0] failed to recover shard]; nested: MapperParsingException[Failed to parse [full_message]]; nested: JsonParseException[Illegal unquoted character ((CTRL-CHAR, code 0)): has to be escaped using backslash to be included in string value at [Source: [B@4bd8d548; line: 1, column: 575]]; ]]

Server2:
[WARN ][cluster.action.shard ] [logserver02] sending failed shard for [graylog2][0], node[lNCk9JoQTIKrT76k2B1pUw], [P], s[INITIALIZING], reason [Failed to start shard, message [IndexShardGatewayRecoveryException[[graylog2][0] failed to recover shard]; nested: MapperParsingException[Failed to parse [message]]; nested: JsonParseException[Illegal unquoted character ((CTRL-CHAR, code 0)): has to be escaped using backslash to be included in string value at [Source: [B@77a202bf; line: 1, column: 79]]; ]]

Regarding the last comment that Igor wrote, do I only have to do the procedure he described on the second server where the "sending" failed, or on the whole cluster?

I'm really a newbie on elasticsearch so sorry if I cannot provide you more professional input.