I had a disk problem on my development laptop and my ES index is only
barely working. Before restoring a [old] snapshot and just moving forward,
I'd like to learn if this can be recovered.
My first sign of trouble were query errors like this:
{
"error": "NoShardAvailableActionException[[twitter_profiles][0] null]",
"status": 503
}
The disk checks out OK now, but the logs below show what happens when the
node starts up.
What's best way to proceed?
Thanks in advance!
The details are here http://pastebin.com/BCXe87Ri, but the high points
are:
[2015-03-26 20:36:55,065][INFO ][node ] [Glob] version[
1.4.2], pid[10652], build[927caff/2014-12-16T14:11:12Z]
[2015-03-26 20:37:03,101][WARN ][indices.cluster ] [Glob]
[twitter_profiles][0] failed to start shard
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException:
[twitter_profiles][0] failed to fetch index version after copying it over
Caused by: org.apache.lucene.index.CorruptIndexException:
[twitter_profiles][0] Preexisting corrupted index
[corrupted_dd6VM9dCQOeajI_AV9COnA] caused by: CorruptIndexException[Invalid
fieldsStream maxPointer (file truncated?): maxPointer=86078308,
length=4294967295]
org.apache.lucene.index.CorruptIndexException: Invalid fieldsStream
maxPointer (file truncated?): maxPointer=86078308, length=4294967295
[2015-03-26 20:37:03,261][WARN ][cluster.action.shard ] [Glob]
[twitter_profiles][0] sending failed shard for [twitter_profiles][0],
node[msa32a5JQHW5aojBitwEqQ], [P], s[INITIALIZING], indexUUID
[1tNwpxI0Rl6M2yldd_l5lw], reason [Failed to start shard, message
[IndexShardGatewayRecoveryException[[twitter_profiles][0] failed to fetch
index version after copying it over]; nested:
CorruptIndexException[[twitter_profiles][0] Preexisting corrupted index
[corrupted_dd6VM9dCQOeajI_AV9COnA] caused by: CorruptIndexException[Invalid
fieldsStream maxPointer (file truncated?): maxPointer=86078308,
length=4294967295]
org.apache.lucene.index.CorruptIndexException: Invalid fieldsStream
maxPointer (file truncated?): maxPointer=86078308, length=4294967295
[2015-03-26 20:37:03,262][WARN ][cluster.action.shard ] [Glob]
[twitter_profiles][0] received shard failed for [twitter_profiles][0],
node[msa32a5JQHW5aojBitwEqQ], [P], s[INITIALIZING], indexUUID
[1tNwpxI0Rl6M2yldd_l5lw], reason [Failed to start shard, message
[IndexShardGatewayRecoveryException[[twitter_profiles][0] failed to fetch
index version after copying it over]; nested:
CorruptIndexException[[twitter_profiles][0] Preexisting corrupted index
[corrupted_dd6VM9dCQOeajI_AV9COnA] caused by: CorruptIndexException[Invalid
fieldsStream maxPointer (file truncated?): maxPointer=86078308,
length=4294967295]
org.apache.lucene.index.CorruptIndexException: Invalid fieldsStream
maxPointer (file truncated?): maxPointer=86078308, length=4294967295
[2015-03-26 20:37:03,262][INFO ][gateway ] [Glob] recovered
[2] indices into cluster_state
[2015-03-26 20:37:17,612][WARN ][indices.cluster ] [Glob]
[twitter_profiles][0] failed to start shard
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException:
[twitter_profiles][0] failed to fetch index version after copying it over
Caused by: org.apache.lucene.index.CorruptIndexException:
[twitter_profiles][0] Preexisting corrupted index
[corrupted_dd6VM9dCQOeajI_AV9COnA] caused by: CorruptIndexException[Invalid
fieldsStream maxPointer (file truncated?): maxPointer=86078308,
length=4294967295]
org.apache.lucene.index.CorruptIndexException: Invalid fieldsStream
maxPointer (file truncated?): maxPointer=86078308, length=4294967295
[2015-03-26 20:37:17,613][WARN ][cluster.action.shard ] [Glob]
[twitter_profiles][0] sending failed shard for [twitter_profiles][0],
node[msa32a5JQHW5aojBitwEqQ], [P], s[INITIALIZING], indexUUID
[1tNwpxI0Rl6M2yldd_l5lw], reason [Failed to start shard, message
[IndexShardGatewayRecoveryException[[twitter_profiles][0] failed to fetch
index version after copying it over]; nested:
CorruptIndexException[[twitter_profiles][0] Preexisting corrupted index
[corrupted_dd6VM9dCQOeajI_AV9COnA] caused by: CorruptIndexException[Invalid
fieldsStream maxPointer (file truncated?): maxPointer=86078308,
length=4294967295]
org.apache.lucene.index.CorruptIndexException: Invalid fieldsStream
maxPointer (file truncated?): maxPointer=86078308, length=4294967295
[2015-03-26 20:37:17,613][WARN ][cluster.action.shard ] [Glob]
[twitter_profiles][0] received shard failed for [twitter_profiles][0],
node[msa32a5JQHW5aojBitwEqQ], [P], s[INITIALIZING], indexUUID
[1tNwpxI0Rl6M2yldd_l5lw], reason [Failed to start shard, message
[IndexShardGatewayRecoveryException[[twitter_profiles][0] failed to fetch
index version after copying it over]; nested:
CorruptIndexException[[twitter_profiles][0] Preexisting corrupted index
[corrupted_dd6VM9dCQOeajI_AV9COnA] caused by: CorruptIndexException[Invalid
fieldsStream maxPointer (file truncated?): maxPointer=86078308,
length=4294967295]
org.apache.lucene.index.CorruptIndexException: Invalid fieldsStream
maxPointer (file truncated?): maxPointer=86078308, length=4294967295
[2015-03-26 20:37:17,719][WARN ][cluster.action.shard ] [Glob]
[twitter_profiles][0] received shard failed for [twitter_profiles][0],
node[msa32a5JQHW5aojBitwEqQ], [P], s[INITIALIZING], indexUUID
[1tNwpxI0Rl6M2yldd_l5lw], reason [master
[Glob][msa32a5JQHW5aojBitwEqQ][mymachine][inet[/192.168.10.101:9300]]
marked shard as initializing, but shard is marked as failed, resend shard
failure]
[2015-03-26 20:37:18,702][WARN ][indices.cluster ] [Glob]
[twitter_profiles][0] failed to start shard
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException:
[twitter_profiles][0] failed to fetch index version after copying it over
Caused by: org.apache.lucene.index.CorruptIndexException:
[twitter_profiles][0] Preexisting corrupted index
[corrupted_dd6VM9dCQOeajI_AV9COnA] caused by: CorruptIndexException[Invalid
fieldsStream maxPointer (file truncated?): maxPointer=86078308,
length=4294967295]
org.apache.lucene.index.CorruptIndexException: Invalid fieldsStream
maxPointer (file truncated?): maxPointer=86078308, length=4294967295
[2015-03-26 20:37:18,703][WARN ][cluster.action.shard ] [Glob]
[twitter_profiles][0] sending failed shard for [twitter_profiles][0],
node[msa32a5JQHW5aojBitwEqQ], [P], s[INITIALIZING], indexUUID
[1tNwpxI0Rl6M2yldd_l5lw], reason [Failed to start shard, message
[IndexShardGatewayRecoveryException[[twitter_profiles][0] failed to fetch
index version after copying it over]; nested:
CorruptIndexException[[twitter_profiles][0] Preexisting corrupted index
[corrupted_dd6VM9dCQOeajI_AV9COnA] caused by: CorruptIndexException[Invalid
fieldsStream maxPointer (file truncated?): maxPointer=86078308,
length=4294967295]
org.apache.lucene.index.CorruptIndexException: Invalid fieldsStream
maxPointer (file truncated?): maxPointer=86078308, length=4294967295
[2015-03-26 20:37:18,703][WARN ][cluster.action.shard ] [Glob]
[twitter_profiles][0] received shard failed for [twitter_profiles][0],
node[msa32a5JQHW5aojBitwEqQ], [P], s[INITIALIZING], indexUUID
[1tNwpxI0Rl6M2yldd_l5lw], reason [Failed to start shard, message
[IndexShardGatewayRecoveryException[[twitter_profiles][0] failed to fetch
index version after copying it over]; nested:
CorruptIndexException[[twitter_profiles][0] Preexisting corrupted index
[corrupted_dd6VM9dCQOeajI_AV9COnA] caused by: CorruptIndexException[Invalid
fieldsStream maxPointer (file truncated?): maxPointer=86078308,
length=4294967295]
org.apache.lucene.index.CorruptIndexException: Invalid fieldsStream
maxPointer (file truncated?): maxPointer=86078308, length=4294967295
[2015-03-26 20:37:18,947][WARN ][indices.cluster ] [Glob]
[twitter_profiles][0] failed to start shard
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException:
[twitter_profiles][0] failed to fetch index version after copying it over
Caused by: org.apache.lucene.index.CorruptIndexException:
[twitter_profiles][0] Preexisting corrupted index
[corrupted_dd6VM9dCQOeajI_AV9COnA] caused by: CorruptIndexException[Invalid
fieldsStream maxPointer (file truncated?): maxPointer=86078308,
length=4294967295]
org.apache.lucene.index.CorruptIndexException: Invalid fieldsStream
maxPointer (file truncated?): maxPointer=86078308, length=4294967295
[2015-03-26 20:37:18,948][WARN ][cluster.action.shard ] [Glob]
[twitter_profiles][0] sending failed shard for [twitter_profiles][0],
node[msa32a5JQHW5aojBitwEqQ], [P], s[INITIALIZING], indexUUID
[1tNwpxI0Rl6M2yldd_l5lw], reason [Failed to start shard, message
[IndexShardGatewayRecoveryException[[twitter_profiles][0] failed to fetch
index version after copying it over]; nested:
CorruptIndexException[[twitter_profiles][0] Preexisting corrupted index
[corrupted_dd6VM9dCQOeajI_AV9COnA] caused by: CorruptIndexException[Invalid
fieldsStream maxPointer (file truncated?): maxPointer=86078308,
length=4294967295]
org.apache.lucene.index.CorruptIndexException: Invalid fieldsStream
maxPointer (file truncated?): maxPointer=86078308, length=4294967295
[2015-03-26 20:37:18,948][WARN ][cluster.action.shard ] [Glob]
[twitter_profiles][0] received shard failed for [twitter_profiles][0],
node[msa32a5JQHW5aojBitwEqQ], [P], s[INITIALIZING], indexUUID
[1tNwpxI0Rl6M2yldd_l5lw], reason [Failed to start shard, message
[IndexShardGatewayRecoveryException[[twitter_profiles][0] failed to fetch
index version after copying it over]; nested:
CorruptIndexException[[twitter_profiles][0] Preexisting corrupted index
[corrupted_dd6VM9dCQOeajI_AV9COnA] caused by: CorruptIndexException[Invalid
fieldsStream maxPointer (file truncated?): maxPointer=86078308,
length=4294967295]
org.apache.lucene.index.CorruptIndexException: Invalid fieldsStream
maxPointer (file truncated?): maxPointer=86078308, length=4294967295
Thoughts?
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/755c0650-9612-4924-9b9c-d52316d99cca%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.