error trying to access river - IndexShardGatewayRecoveryException

So, I my server ran out of space (which is possibly what started this
problem), but now I can't get my couchdb river going again.

I've deleted my river and when I try and recreate it I get this error:
{"error":"UnavailableShardsException[[_river][0] [2] shardIt, [0]
active : Timeout waiting for [1m], request: index
{[_river][syncrswim][_meta], source[{"type" :
"couchdb","couchdb" : {"host" : "localhost","port" :
5984,"db" :"syncrswim","filter" : null},"index" : {"index" :
"syncrswim", "type" : "syncrswim", "bulk_size" :
"100", "bulk_timeout" : "10ms" }}]}]","status":503}

before it was removed I was seeing this in the log

[2012-08-11 14:45:37,883][WARN ][indices.cluster ] [Golubev,
Mikula] [_river][0] failed to start shard
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException:
[_river][0] failed to recover shard
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:216)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:177)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
Caused by: java.lang.StringIndexOutOfBoundsException: String index out
of range: 0
at java.lang.String.charAt(String.java:694)
at org.elasticsearch.index.mapper.MapperService.add(MapperService.java:182)
at org.elasticsearch.index.mapper.MapperService.add(MapperService.java:174)
at org.elasticsearch.index.mapper.MapperService.documentMapperWithAutoCreate(MapperService.java:300)
at org.elasticsearch.index.shard.service.InternalIndexShard.prepareIndex(InternalIndexShard.java:310)
at org.elasticsearch.index.shard.service.InternalIndexShard.performRecoveryOperation(InternalIndexShard.java:624)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:211)
... 4 more
[2012-08-11 14:45:37,890][WARN ][cluster.action.shard ] [Golubev,
Mikula] sending failed shard for [_river][0],
node[I0pOvJWbQoSVJDqQb3ZZ2A], [P], s[INITIALIZING], reason [Failed to
start shard, message [IndexShardGatewayRecoveryException[[_river][0]
failed to recover shard]; nested:
StringIndexOutOfBoundsException[String index out of range: 0]; ]]
[2012-08-11 14:45:37,890][WARN ][cluster.action.shard ] [Golubev,
Mikula] received shard failed for [_river][0],
node[I0pOvJWbQoSVJDqQb3ZZ2A], [P], s[INITIALIZING], reason [Failed to
start shard, message [IndexShardGatewayRecoveryException[[_river][0]
failed to recover shard]; nested:
StringIndexOutOfBoundsException[String index out of range: 0]; ]]

Any thoughts on how to recover from this? I'm still in development
mode so if need be I can wipe out my data, just not sure how to do
this in a safe way.

-warner

--

This will be fixed in 0.19.9 where there is better handling of corrupt translog. Now, you can simply delete the "data" location of ES if you want to start fresh, or, in case you would have run this in production, and you had replicas allocated, those replicas would have been used for recovery automatically.

On Aug 12, 2012, at 12:24 AM, Warner Onstine warnero@gmail.com wrote:

So, I my server ran out of space (which is possibly what started this
problem), but now I can't get my couchdb river going again.

I've deleted my river and when I try and recreate it I get this error:
{"error":"UnavailableShardsException[[_river][0] [2] shardIt, [0]
active : Timeout waiting for [1m], request: index
{[_river][syncrswim][_meta], source[{"type" :
"couchdb","couchdb" : {"host" : "localhost","port" :
5984,"db" :"syncrswim","filter" : null},"index" : {"index" :
"syncrswim", "type" : "syncrswim", "bulk_size" :
"100", "bulk_timeout" : "10ms" }}]}]","status":503}

before it was removed I was seeing this in the log

[2012-08-11 14:45:37,883][WARN ][indices.cluster ] [Golubev,
Mikula] [_river][0] failed to start shard
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException:
[_river][0] failed to recover shard
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:216)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:177)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
Caused by: java.lang.StringIndexOutOfBoundsException: String index out
of range: 0
at java.lang.String.charAt(String.java:694)
at org.elasticsearch.index.mapper.MapperService.add(MapperService.java:182)
at org.elasticsearch.index.mapper.MapperService.add(MapperService.java:174)
at org.elasticsearch.index.mapper.MapperService.documentMapperWithAutoCreate(MapperService.java:300)
at org.elasticsearch.index.shard.service.InternalIndexShard.prepareIndex(InternalIndexShard.java:310)
at org.elasticsearch.index.shard.service.InternalIndexShard.performRecoveryOperation(InternalIndexShard.java:624)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:211)
... 4 more
[2012-08-11 14:45:37,890][WARN ][cluster.action.shard ] [Golubev,
Mikula] sending failed shard for [_river][0],
node[I0pOvJWbQoSVJDqQb3ZZ2A], [P], s[INITIALIZING], reason [Failed to
start shard, message [IndexShardGatewayRecoveryException[[_river][0]
failed to recover shard]; nested:
StringIndexOutOfBoundsException[String index out of range: 0]; ]]
[2012-08-11 14:45:37,890][WARN ][cluster.action.shard ] [Golubev,
Mikula] received shard failed for [_river][0],
node[I0pOvJWbQoSVJDqQb3ZZ2A], [P], s[INITIALIZING], reason [Failed to
start shard, message [IndexShardGatewayRecoveryException[[_river][0]
failed to recover shard]; nested:
StringIndexOutOfBoundsException[String index out of range: 0]; ]]

Any thoughts on how to recover from this? I'm still in development
mode so if need be I can wipe out my data, just not sure how to do
this in a safe way.

-warner

--

--