Endless recovery

Elastic Stack Elasticsearch
1

Recovery is failing due to an error in Source field. Basically end up
with nonstop logging of a JSON parsing in the Source field. So we
somehow placed data in with a bad character in the JSON? Is that not
checked when updating, but only on recovery? Logging obviously not
saying more about the failing item.

[2011-11-07 21:04:46,525][WARN ][cluster.action.shard ] [Banshee]
received shard failed for [media][14], node[aTT5vuDsSheosUuFyhij8w],
[R], s[INITIALIZING], reason [Failed to start shard, message
[RecoveryFailedException[Index Shard [media][14]: Recovery failed from
[Andromeda][_Swg_4NsRru28Ppp3_1mFg][inet[/10.201.214.240:9300]] into
[White Fang][aTT5vuDsSheosUuFyhij8w][inet[/10.96.154.59:9300]]];
nested: RemoteTransportException[[Andromeda][inet[/
10.201.214.240:9300]][index/shard/recovery/startRecovery]]; nested:
RecoveryEngineException[[media][14] Phase[2] Execution failed];
nested: RemoteTransportException[[White Fang][inet[/10.96.154.59:9300]]
[index/shard/recovery/translogOps]]; nested:
MapperParsingException[Failed to parse [extras.aid]]; nested:
JsonParseException[Illegal unquoted character ((CTRL-CHAR, code 0)):
has to be escaped using backslash to be included in string value
at [Source: [B@2511ad41; line: 1, column: 802]]; ]]
[2011-11-07 21:04:46,645][WARN ][cluster.action.shard ] [Banshee]
received shard failed for [media][14], node[BVIFqg9sTjSwcpQg9-AqlQ],
[R], s[INITIALIZING], reason [Failed to start shard, message
[RecoveryFailedException[Index Shard [media][14]: Recovery failed from
[Andromeda][_Swg_4NsRru28Ppp3_1mFg][inet[/10.201.214.240:9300]] into
[Gorr][BVIFqg9sTjSwcpQg9-AqlQ][inet[/10.72.57.189:9300]]]; nested:
RemoteTransportException[[Andromeda][inet[/10.201.214.240:9300]][index/
shard/recovery/startRecovery]]; nested: RecoveryEngineException[[media]
[14] Phase[2] Execution failed]; nested:
RemoteTransportException[[Gorr][inet[/10.72.57.189:9300]][index/shard/
recovery/translogOps]]; nested: MapperParsingException[Failed to parse
[extras.aid]]; nested: JsonParseException[Illegal unquoted character
((CTRL-CHAR, code 0)): has to be escaped using backslash to be
included in string value
at [Source: [B@7d5b668; line: 1, column: 802]]; ]]
[2011-11-07 21:04:46,842][WARN ][cluster.action.shard ] [Banshee]
received shard failed for [media][14], node[aTT5vuDsSheosUuFyhij8w],
[R], s[INITIALIZING], reason [Failed to start shard, message
[RecoveryFailedException[Index Shard [media][14]: Recovery failed from
[Andromeda][_Swg_4NsRru28Ppp3_1mFg][inet[/10.201.214.240:9300]] into
[White Fang][aTT5vuDsSheosUuFyhij8w][inet[/10.96.154.59:9300]]];
nested: RemoteTransportException[[Andromeda][inet[/
10.201.214.240:9300]][index/shard/recovery/startRecovery]]; nested:
RecoveryEngineException[[media][14] Phase[2] Execution failed];
nested: RemoteTransportException[[White Fang][inet[/10.96.154.59:9300]]
[index/shard/recovery/translogOps]]; nested:
MapperParsingException[Failed to parse [extras.aid]]; nested:
JsonParseException[Illegal unquoted character ((CTRL-CHAR, code 0)):
has to be escaped using backslash to be included in string value
at [Source: [B@4a8265ab; line: 1, column: 802]]; ]]
[2011-11-07 21:04:46,939][WARN ][cluster.action.shard ] [Banshee]
received shard failed for [media][14], node[BVIFqg9sTjSwcpQg9-AqlQ],
[R], s[INITIALIZING], reason [Failed to start shard, message
[RecoveryFailedException[Index Shard [media][14]: Recovery failed from
[Andromeda][_Swg_4NsRru28Ppp3_1mFg][inet[/10.201.214.240:9300]] into
[Gorr][BVIFqg9sTjSwcpQg9-AqlQ][inet[/10.72.57.189:9300]]]; nested:
RemoteTransportException[[Andromeda][inet[/10.201.214.240:9300]][index/
shard/recovery/startRecovery]]; nested: RecoveryEngineException[[media]
[14] Phase[2] Execution failed]; nested:
RemoteTransportException[[Gorr][inet[/10.72.57.189:9300]][index/shard/
recovery/translogOps]]; nested: MapperParsingException[Failed to parse
[extras.aid]]; nested: JsonParseException[Illegal unquoted character
((CTRL-CHAR, code 0)): has to be escaped using backslash to be
included in string value
at [Source: [B@1a8c5371; line: 1, column: 802]]; ]]
[2011-11-07 21:04:47,097][WARN ][cluster.action.shard ] [Banshee]
received shard failed for [media][14], node[aTT5vuDsSheosUuFyhij8w],
[R], s[INITIALIZING], reason [Failed to start shard, message
[RecoveryFailedException[Index Shard [media][14]: Recovery failed from
[Andromeda][_Swg_4NsRru28Ppp3_1mFg][inet[/10.201.214.240:9300]] into
[White Fang][aTT5vuDsSheosUuFyhij8w][inet[/10.96.154.59:9300]]];
nested: RemoteTransportException[[Andromeda][inet[/
10.201.214.240:9300]][index/shard/recovery/startRecovery]]; nested:
RecoveryEngineException[[media][14] Phase[2] Execution failed];
nested: RemoteTransportException[[White Fang][inet[/10.96.154.59:9300]]
[index/shard/recovery/translogOps]]; nested:
MapperParsingException[Failed to parse [extras.aid]]; nested:
JsonParseException[Illegal unquoted character ((CTRL-CHAR, code 0)):
has to be escaped using backslash to be included in string value
at [Source: [B@4a59609f; line: 1, column: 802]]; ]]
[2011-11-07 21:04:47,229][WARN ][cluster.action.shard ] [Banshee]
received shard failed for [media][14], node[BVIFqg9sTjSwcpQg9-AqlQ],
[R], s[INITIALIZING], reason [Failed to start shard, message
[RecoveryFailedException[Index Shard [media][14]: Recovery failed from
[Andromeda][_Swg_4NsRru28Ppp3_1mFg][inet[/10.201.214.240:9300]] into
[Gorr][BVIFqg9sTjSwcpQg9-AqlQ][inet[/10.72.57.189:9300]]]; nested:
RemoteTransportException[[Andromeda][inet[/10.201.214.240:9300]][index/
shard/recovery/startRecovery]]; nested: RecoveryEngineException[[media]
[14] Phase[2] Execution failed]; nested:
RemoteTransportException[[Gorr][inet[/10.72.57.189:9300]][index/shard/
recovery/translogOps]]; nested: MapperParsingException[Failed to parse
[extras.aid]]; nested: JsonParseException[Illegal unquoted character
((CTRL-CHAR, code 0)): has to be escaped using backslash to be
included in string value

2

It is checked when indexing, strange... . You can get around it by sending
a flush request to the index, but I am not sure what the state of the
document that causes the failure is (the _source might not be a valid json).

On Mon, Nov 7, 2011 at 11:06 PM, jminard jayson.minard@gmail.com wrote:

Recovery is failing due to an error in Source field. Basically end up
with nonstop logging of a JSON parsing in the Source field. So we
somehow placed data in with a bad character in the JSON? Is that not
checked when updating, but only on recovery? Logging obviously not
saying more about the failing item.

[2011-11-07 21:04:46,525][WARN ][cluster.action.shard ] [Banshee]
received shard failed for [media][14], node[aTT5vuDsSheosUuFyhij8w],
[R], s[INITIALIZING], reason [Failed to start shard, message
[RecoveryFailedException[Index Shard [media][14]: Recovery failed from
[Andromeda][_Swg_4NsRru28Ppp3_1mFg][inet[/10.201.214.240:9300]] into
[White Fang][aTT5vuDsSheosUuFyhij8w][inet[/10.96.154.59:9300]]];
nested: RemoteTransportException[[Andromeda][inet[/
10.201.214.240:9300]][index/shard/recovery/startRecovery]]; nested:
RecoveryEngineException[[media][14] Phase[2] Execution failed];
nested: RemoteTransportException[[White Fang][inet[/10.96.154.59:9300]]
[index/shard/recovery/translogOps]]; nested:
MapperParsingException[Failed to parse [extras.aid]]; nested:
JsonParseException[Illegal unquoted character ((CTRL-CHAR, code 0)):
has to be escaped using backslash to be included in string value
at [Source: [B@2511ad41; line: 1, column: 802]]; ]]
[2011-11-07 21:04:46,645][WARN ][cluster.action.shard ] [Banshee]
received shard failed for [media][14], node[BVIFqg9sTjSwcpQg9-AqlQ],
[R], s[INITIALIZING], reason [Failed to start shard, message
[RecoveryFailedException[Index Shard [media][14]: Recovery failed from
[Andromeda][_Swg_4NsRru28Ppp3_1mFg][inet[/10.201.214.240:9300]] into
[Gorr][BVIFqg9sTjSwcpQg9-AqlQ][inet[/10.72.57.189:9300]]]; nested:
RemoteTransportException[[Andromeda][inet[/10.201.214.240:9300]][index/
shard/recovery/startRecovery]]; nested: RecoveryEngineException[[media]
[14] Phase[2] Execution failed]; nested:
RemoteTransportException[[Gorr][inet[/10.72.57.189:9300]][index/shard/
recovery/translogOps]]; nested: MapperParsingException[Failed to parse
[extras.aid]]; nested: JsonParseException[Illegal unquoted character
((CTRL-CHAR, code 0)): has to be escaped using backslash to be
included in string value
at [Source: [B@7d5b668; line: 1, column: 802]]; ]]
[2011-11-07 21:04:46,842][WARN ][cluster.action.shard ] [Banshee]
received shard failed for [media][14], node[aTT5vuDsSheosUuFyhij8w],
[R], s[INITIALIZING], reason [Failed to start shard, message
[RecoveryFailedException[Index Shard [media][14]: Recovery failed from
[Andromeda][_Swg_4NsRru28Ppp3_1mFg][inet[/10.201.214.240:9300]] into
[White Fang][aTT5vuDsSheosUuFyhij8w][inet[/10.96.154.59:9300]]];
nested: RemoteTransportException[[Andromeda][inet[/
10.201.214.240:9300]][index/shard/recovery/startRecovery]]; nested:
RecoveryEngineException[[media][14] Phase[2] Execution failed];
nested: RemoteTransportException[[White Fang][inet[/10.96.154.59:9300]]
[index/shard/recovery/translogOps]]; nested:
MapperParsingException[Failed to parse [extras.aid]]; nested:
JsonParseException[Illegal unquoted character ((CTRL-CHAR, code 0)):
has to be escaped using backslash to be included in string value
at [Source: [B@4a8265ab; line: 1, column: 802]]; ]]
[2011-11-07 21:04:46,939][WARN ][cluster.action.shard ] [Banshee]
received shard failed for [media][14], node[BVIFqg9sTjSwcpQg9-AqlQ],
[R], s[INITIALIZING], reason [Failed to start shard, message
[RecoveryFailedException[Index Shard [media][14]: Recovery failed from
[Andromeda][_Swg_4NsRru28Ppp3_1mFg][inet[/10.201.214.240:9300]] into
[Gorr][BVIFqg9sTjSwcpQg9-AqlQ][inet[/10.72.57.189:9300]]]; nested:
RemoteTransportException[[Andromeda][inet[/10.201.214.240:9300]][index/
shard/recovery/startRecovery]]; nested: RecoveryEngineException[[media]
[14] Phase[2] Execution failed]; nested:
RemoteTransportException[[Gorr][inet[/10.72.57.189:9300]][index/shard/
recovery/translogOps]]; nested: MapperParsingException[Failed to parse
[extras.aid]]; nested: JsonParseException[Illegal unquoted character
((CTRL-CHAR, code 0)): has to be escaped using backslash to be
included in string value
at [Source: [B@1a8c5371; line: 1, column: 802]]; ]]
[2011-11-07 21:04:47,097][WARN ][cluster.action.shard ] [Banshee]
received shard failed for [media][14], node[aTT5vuDsSheosUuFyhij8w],
[R], s[INITIALIZING], reason [Failed to start shard, message
[RecoveryFailedException[Index Shard [media][14]: Recovery failed from
[Andromeda][_Swg_4NsRru28Ppp3_1mFg][inet[/10.201.214.240:9300]] into
[White Fang][aTT5vuDsSheosUuFyhij8w][inet[/10.96.154.59:9300]]];
nested: RemoteTransportException[[Andromeda][inet[/
10.201.214.240:9300]][index/shard/recovery/startRecovery]]; nested:
RecoveryEngineException[[media][14] Phase[2] Execution failed];
nested: RemoteTransportException[[White Fang][inet[/10.96.154.59:9300]]
[index/shard/recovery/translogOps]]; nested:
MapperParsingException[Failed to parse [extras.aid]]; nested:
JsonParseException[Illegal unquoted character ((CTRL-CHAR, code 0)):
has to be escaped using backslash to be included in string value
at [Source: [B@4a59609f; line: 1, column: 802]]; ]]
[2011-11-07 21:04:47,229][WARN ][cluster.action.shard ] [Banshee]
received shard failed for [media][14], node[BVIFqg9sTjSwcpQg9-AqlQ],
[R], s[INITIALIZING], reason [Failed to start shard, message
[RecoveryFailedException[Index Shard [media][14]: Recovery failed from
[Andromeda][_Swg_4NsRru28Ppp3_1mFg][inet[/10.201.214.240:9300]] into
[Gorr][BVIFqg9sTjSwcpQg9-AqlQ][inet[/10.72.57.189:9300]]]; nested:
RemoteTransportException[[Andromeda][inet[/10.201.214.240:9300]][index/
shard/recovery/startRecovery]]; nested: RecoveryEngineException[[media]
[14] Phase[2] Execution failed]; nested:
RemoteTransportException[[Gorr][inet[/10.72.57.189:9300]][index/shard/
recovery/translogOps]]; nested: MapperParsingException[Failed to parse
[extras.aid]]; nested: JsonParseException[Illegal unquoted character
((CTRL-CHAR, code 0)): has to be escaped using backslash to be
included in string value