# Endless recovery **URL:** \ **Category:** Elasticsearch\ **Created:** [November 7, 2011, 9:06pm UTC](https://discuss.elastic.co/t/endless-recovery/5800 "2011-11-07T21:06:45Z")\ **Posts on this page:** 3\ **Page:** 1
**Author:** ![jminard](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jminard/32/1848_2.png) [@jminard](https://discuss.elastic.co/u/jminard)\ **Post date:** [November 7, 2011, 9:06pm UTC](https://discuss.elastic.co/t/endless-recovery/5800/1 "2011-11-07T21:06:45Z")
Recovery is failing due to an error in Source field. Basically end up with nonstop logging of a JSON parsing in the Source field. So we somehow placed data in with a bad character in the JSON? Is that not checked when updating, but only on recovery? Logging obviously not saying more about the failing item. [2011-11-07 21:04:46,525][WARN][cluster.action.shard] [Banshee] received shard failed for [media][14], node[aTT5vuDsSheosUuFyhij8w], [R], s[INITIALIZING], reason [Failed to start shard, message [RecoveryFailedException[Index Shard [media][14]: Recovery failed from [Andromeda][\_Swg\_4NsRru28Ppp3\_1mFg][inet[/10.201.214.240:9300]] into [White Fang][aTT5vuDsSheosUuFyhij8w][inet[/10.96.154.59:9300]]]; nested: RemoteTransportException[[Andromeda][inet[/ 10.201.214.240:9300]][index/shard/recovery/startRecovery]]; nested: RecoveryEngineException[[media][14] Phase[2] Execution failed]; nested: RemoteTransportException[[White Fang][inet[/10.96.154.59:9300]] [index/shard/recovery/translogOps]]; nested: MapperParsingException[Failed to parse [extras.aid]]; nested: JsonParseException[Illegal unquoted character ((CTRL-CHAR, code 0)): has to be escaped using backslash to be included in string value at [Source: [B@2511ad41; line: 1, column: 802]]; ]] [2011-11-07 21:04:46,645][WARN][cluster.action.shard] [Banshee] received shard failed for [media][14], node[BVIFqg9sTjSwcpQg9-AqlQ], [R], s[INITIALIZING], reason [Failed to start shard, message [RecoveryFailedException[Index Shard [media][14]: Recovery failed from [Andromeda][\_Swg\_4NsRru28Ppp3\_1mFg][inet[/10.201.214.240:9300]] into [Gorr][BVIFqg9sTjSwcpQg9-AqlQ][inet[/10.72.57.189:9300]]]; nested: RemoteTransportException[[Andromeda][inet[/10.201.214.240:9300]][index/ shard/recovery/startRecovery]]; nested: RecoveryEngineException[[media] [14] Phase[2] Execution failed]; nested: RemoteTransportException[[Gorr][inet[/10.72.57.189:9300]][index/shard/ recovery/translogOps]]; nested: MapperParsingException[Failed to parse [extras.aid]]; nested: JsonParseException[Illegal unquoted character ((CTRL-CHAR, code 0)): has to be escaped using backslash to be included in string value at [Source: [B@7d5b668; line: 1, column: 802]]; ]] [2011-11-07 21:04:46,842][WARN][cluster.action.shard] [Banshee] received shard failed for [media][14], node[aTT5vuDsSheosUuFyhij8w], [R], s[INITIALIZING], reason [Failed to start shard, message [RecoveryFailedException[Index Shard [media][14]: Recovery failed from [Andromeda][\_Swg\_4NsRru28Ppp3\_1mFg][inet[/10.201.214.240:9300]] into [White Fang][aTT5vuDsSheosUuFyhij8w][inet[/10.96.154.59:9300]]]; nested: RemoteTransportException[[Andromeda][inet[/ 10.201.214.240:9300]][index/shard/recovery/startRecovery]]; nested: RecoveryEngineException[[media][14] Phase[2] Execution failed]; nested: RemoteTransportException[[White Fang][inet[/10.96.154.59:9300]] [index/shard/recovery/translogOps]]; nested: MapperParsingException[Failed to parse [extras.aid]]; nested: JsonParseException[Illegal unquoted character ((CTRL-CHAR, code 0)): has to be escaped using backslash to be included in string value at [Source: [B@4a8265ab; line: 1, column: 802]]; ]] [2011-11-07 21:04:46,939][WARN][cluster.action.shard] [Banshee] received shard failed for [media][14], node[BVIFqg9sTjSwcpQg9-AqlQ], [R], s[INITIALIZING], reason [Failed to start shard, message [RecoveryFailedException[Index Shard [media][14]: Recovery failed from [Andromeda][\_Swg\_4NsRru28Ppp3\_1mFg][inet[/10.201.214.240:9300]] into [Gorr][BVIFqg9sTjSwcpQg9-AqlQ][inet[/10.72.57.189:9300]]]; nested: RemoteTransportException[[Andromeda][inet[/10.201.214.240:9300]][index/ shard/recovery/startRecovery]]; nested: RecoveryEngineException[[media] [14] Phase[2] Execution failed]; nested: RemoteTransportException[[Gorr][inet[/10.72.57.189:9300]][index/shard/ recovery/translogOps]]; nested: MapperParsingException[Failed to parse [extras.aid]]; nested: JsonParseException[Illegal unquoted character ((CTRL-CHAR, code 0)): has to be escaped using backslash to be included in string value at [Source: [B@1a8c5371; line: 1, column: 802]]; ]] [2011-11-07 21:04:47,097][WARN][cluster.action.shard] [Banshee] received shard failed for [media][14], node[aTT5vuDsSheosUuFyhij8w], [R], s[INITIALIZING], reason [Failed to start shard, message [RecoveryFailedException[Index Shard [media][14]: Recovery failed from [Andromeda][\_Swg\_4NsRru28Ppp3\_1mFg][inet[/10.201.214.240:9300]] into [White Fang][aTT5vuDsSheosUuFyhij8w][inet[/10.96.154.59:9300]]]; nested: RemoteTransportException[[Andromeda][inet[/ 10.201.214.240:9300]][index/shard/recovery/startRecovery]]; nested: RecoveryEngineException[[media][14] Phase[2] Execution failed]; nested: RemoteTransportException[[White Fang][inet[/10.96.154.59:9300]] [index/shard/recovery/translogOps]]; nested: MapperParsingException[Failed to parse [extras.aid]]; nested: JsonParseException[Illegal unquoted character ((CTRL-CHAR, code 0)): has to be escaped using backslash to be included in string value at [Source: [B@4a59609f; line: 1, column: 802]]; ]] [2011-11-07 21:04:47,229][WARN][cluster.action.shard] [Banshee] received shard failed for [media][14], node[BVIFqg9sTjSwcpQg9-AqlQ], [R], s[INITIALIZING], reason [Failed to start shard, message [RecoveryFailedException[Index Shard [media][14]: Recovery failed from [Andromeda][\_Swg\_4NsRru28Ppp3\_1mFg][inet[/10.201.214.240:9300]] into [Gorr][BVIFqg9sTjSwcpQg9-AqlQ][inet[/10.72.57.189:9300]]]; nested: RemoteTransportException[[Andromeda][inet[/10.201.214.240:9300]][index/ shard/recovery/startRecovery]]; nested: RecoveryEngineException[[media] [14] Phase[2] Execution failed]; nested: RemoteTransportException[[Gorr][inet[/10.72.57.189:9300]][index/shard/ recovery/translogOps]]; nested: MapperParsingException[Failed to parse [extras.aid]]; nested: JsonParseException[Illegal unquoted character ((CTRL-CHAR, code 0)): has to be escaped using backslash to be included in string value ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [November 9, 2011, 7:05am UTC](https://discuss.elastic.co/t/endless-recovery/5800/2 "2011-11-09T07:05:33Z")
It is checked when indexing, strange... . You can get around it by sending a flush request to the index, but I am not sure what the state of the document that causes the failure is (the \_source might not be a valid json). On Mon, Nov 7, 2011 at 11:06 PM, jminard [jayson.minard@gmail.com](mailto:jayson.minard@gmail.com) wrote: > Recovery is failing due to an error in Source field. Basically end up > with nonstop logging of a JSON parsing in the Source field. So we > somehow placed data in with a bad character in the JSON? Is that not > checked when updating, but only on recovery? Logging obviously not > saying more about the failing item. > > [2011-11-07 21:04:46,525][WARN][cluster.action.shard] [Banshee] > received shard failed for [media][14], node[aTT5vuDsSheosUuFyhij8w], > [R], s[INITIALIZING], reason [Failed to start shard, message > [RecoveryFailedException[Index Shard [media][14]: Recovery failed from > [Andromeda][\_Swg\_4NsRru28Ppp3\_1mFg][inet[/10.201.214.240:9300]] into > [White Fang][aTT5vuDsSheosUuFyhij8w][inet[/10.96.154.59:9300]]]; > nested: RemoteTransportException[[Andromeda][inet[/ > 10.201.214.240:9300]][index/shard/recovery/startRecovery]]; nested: > RecoveryEngineException[[media][14] Phase[2] Execution failed]; > nested: RemoteTransportException[[White Fang][inet[/10.96.154.59:9300]] > [index/shard/recovery/translogOps]]; nested: > MapperParsingException[Failed to parse [extras.aid]]; nested: > JsonParseException[Illegal unquoted character ((CTRL-CHAR, code 0)): > has to be escaped using backslash to be included in string value > at [Source: [B@2511ad41; line: 1, column: 802]]; ]] > [2011-11-07 21:04:46,645][WARN][cluster.action.shard] [Banshee] > received shard failed for [media][14], node[BVIFqg9sTjSwcpQg9-AqlQ], > [R], s[INITIALIZING], reason [Failed to start shard, message > [RecoveryFailedException[Index Shard [media][14]: Recovery failed from > [Andromeda][\_Swg\_4NsRru28Ppp3\_1mFg][inet[/10.201.214.240:9300]] into > [Gorr][BVIFqg9sTjSwcpQg9-AqlQ][inet[/10.72.57.189:9300]]]; nested: > RemoteTransportException[[Andromeda][inet[/10.201.214.240:9300]][index/ > shard/recovery/startRecovery]]; nested: RecoveryEngineException[[media] > [14] Phase[2] Execution failed]; nested: > RemoteTransportException[[Gorr][inet[/10.72.57.189:9300]][index/shard/ > recovery/translogOps]]; nested: MapperParsingException[Failed to parse > [extras.aid]]; nested: JsonParseException[Illegal unquoted character > ((CTRL-CHAR, code 0)): has to be escaped using backslash to be > included in string value > at [Source: [B@7d5b668; line: 1, column: 802]]; ]] > [2011-11-07 21:04:46,842][WARN][cluster.action.shard] [Banshee] > received shard failed for [media][14], node[aTT5vuDsSheosUuFyhij8w], > [R], s[INITIALIZING], reason [Failed to start shard, message > [RecoveryFailedException[Index Shard [media][14]: Recovery failed from > [Andromeda][\_Swg\_4NsRru28Ppp3\_1mFg][inet[/10.201.214.240:9300]] into > [White Fang][aTT5vuDsSheosUuFyhij8w][inet[/10.96.154.59:9300]]]; > nested: RemoteTransportException[[Andromeda][inet[/ > 10.201.214.240:9300]][index/shard/recovery/startRecovery]]; nested: > RecoveryEngineException[[media][14] Phase[2] Execution failed]; > nested: RemoteTransportException[[White Fang][inet[/10.96.154.59:9300]] > [index/shard/recovery/translogOps]]; nested: > MapperParsingException[Failed to parse [extras.aid]]; nested: > JsonParseException[Illegal unquoted character ((CTRL-CHAR, code 0)): > has to be escaped using backslash to be included in string value > at [Source: [B@4a8265ab; line: 1, column: 802]]; ]] > [2011-11-07 21:04:46,939][WARN][cluster.action.shard] [Banshee] > received shard failed for [media][14], node[BVIFqg9sTjSwcpQg9-AqlQ], > [R], s[INITIALIZING], reason [Failed to start shard, message > [RecoveryFailedException[Index Shard [media][14]: Recovery failed from > [Andromeda][\_Swg\_4NsRru28Ppp3\_1mFg][inet[/10.201.214.240:9300]] into > [Gorr][BVIFqg9sTjSwcpQg9-AqlQ][inet[/10.72.57.189:9300]]]; nested: > RemoteTransportException[[Andromeda][inet[/10.201.214.240:9300]][index/ > shard/recovery/startRecovery]]; nested: RecoveryEngineException[[media] > [14] Phase[2] Execution failed]; nested: > RemoteTransportException[[Gorr][inet[/10.72.57.189:9300]][index/shard/ > recovery/translogOps]]; nested: MapperParsingException[Failed to parse > [extras.aid]]; nested: JsonParseException[Illegal unquoted character > ((CTRL-CHAR, code 0)): has to be escaped using backslash to be > included in string value > at [Source: [B@1a8c5371; line: 1, column: 802]]; ]] > [2011-11-07 21:04:47,097][WARN][cluster.action.shard] [Banshee] > received shard failed for [media][14], node[aTT5vuDsSheosUuFyhij8w], > [R], s[INITIALIZING], reason [Failed to start shard, message > [RecoveryFailedException[Index Shard [media][14]: Recovery failed from > [Andromeda][\_Swg\_4NsRru28Ppp3\_1mFg][inet[/10.201.214.240:9300]] into > [White Fang][aTT5vuDsSheosUuFyhij8w][inet[/10.96.154.59:9300]]]; > nested: RemoteTransportException[[Andromeda][inet[/ > 10.201.214.240:9300]][index/shard/recovery/startRecovery]]; nested: > RecoveryEngineException[[media][14] Phase[2] Execution failed]; > nested: RemoteTransportException[[White Fang][inet[/10.96.154.59:9300]] > [index/shard/recovery/translogOps]]; nested: > MapperParsingException[Failed to parse [extras.aid]]; nested: > JsonParseException[Illegal unquoted character ((CTRL-CHAR, code 0)): > has to be escaped using backslash to be included in string value > at [Source: [B@4a59609f; line: 1, column: 802]]; ]] > [2011-11-07 21:04:47,229][WARN][cluster.action.shard] [Banshee] > received shard failed for [media][14], node[BVIFqg9sTjSwcpQg9-AqlQ], > [R], s[INITIALIZING], reason [Failed to start shard, message > [RecoveryFailedException[Index Shard [media][14]: Recovery failed from > [Andromeda][\_Swg\_4NsRru28Ppp3\_1mFg][inet[/10.201.214.240:9300]] into > [Gorr][BVIFqg9sTjSwcpQg9-AqlQ][inet[/10.72.57.189:9300]]]; nested: > RemoteTransportException[[Andromeda][inet[/10.201.214.240:9300]][index/ > shard/recovery/startRecovery]]; nested: RecoveryEngineException[[media] > [14] Phase[2] Execution failed]; nested: > RemoteTransportException[[Gorr][inet[/10.72.57.189:9300]][index/shard/ > recovery/translogOps]]; nested: MapperParsingException[Failed to parse > [extras.aid]]; nested: JsonParseException[Illegal unquoted character > ((CTRL-CHAR, code 0)): has to be escaped using backslash to be > included in string value ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 3:49am UTC](https://discuss.elastic.co/t/endless-recovery/5800/3 "2017-07-06T03:49:24Z")