How to Cleanly Recover After Crash

Didjit · July 22, 2014, 2:29pm

I've ran into some issues with Elasticsearch crashing on my windows box
(most likely my fault). Anyway, is there clean steps to brute force
recover? Something like, delete indexes, issue rebuild, wait .....

I do have the dev version of Marvel if there are tools in there.

Any help is appreciated. Thank you!

Chris

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/a45fff50-7c85-494f-a5ad-5c93892fa40f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Didjit · July 22, 2014, 5:10pm

I keep getting these messages over and over. Any ideas how to repair?

Thank you.

[2014-07-22 13:08:43,966][WARN ][indices.cluster ] [Angelo
Unuscione] [.marvel-2014.07.19][0] failed to start shard
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException:
[.marvel-2014.07.19][0] failed to recover shard
at
org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:241)
at
org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:132)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
Source)
at java.lang.Thread.run(Unknown Source)
Caused by: org.elasticsearch.ElasticsearchIllegalArgumentException: No
version type match [104]
at
org.elasticsearch.index.VersionType.fromValue(VersionType.java:307)
at
org.elasticsearch.index.translog.Translog$Create.readFrom(Translog.java:363)
at
org.elasticsearch.index.translog.TranslogStreams.readTranslogOperation(TranslogStreams.java:52)
at
org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:218)
... 4 more
[2014-07-22 13:08:46,126][WARN ][cluster.action.shard ] [Angelo
Unuscione] [.marvel-2014.07.20][0] sending failed shard for
[.marvel-2014.07.20][0], node[ifz3-S9fTgayreyuIXzHGA], [P],
s[INITIALIZING], indexUUID [B1PAAjUHQGilk8T8nQzTbQ], reason [Failed to
start shard, message
[IndexShardGatewayRecoveryException[[.marvel-2014.07.20][0] failed to
recover shard]; nested: ElasticsearchIllegalArgumentException[No version
type match [104]]; ]]
[

On Tuesday, July 22, 2014 10:29:53 AM UTC-4, Didjit wrote:

I've ran into some issues with Elasticsearch crashing on my windows box
(most likely my fault). Anyway, is there clean steps to brute force
recover? Something like, delete indexes, issue rebuild, wait .....

I do have the dev version of Marvel if there are tools in there.

Any help is appreciated. Thank you!

Chris

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/ba92816f-3cc1-48b2-b56b-2f2e299303aa%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

warkolm · July 22, 2014, 10:16pm

It might be easier to just delete the data and reindex if you can (apart
from marvel).

There are a few lucene tools that you could run, but that's pretty
advanced, there isn't anything within ES itself.

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: markw@campaignmonitor.com
web: www.campaignmonitor.com

On 23 July 2014 03:10, Didjit didjit86@gmail.com wrote:

I keep getting these messages over and over. Any ideas how to repair?

Thank you.

[2014-07-22 13:08:43,966][WARN ][indices.cluster ] [Angelo
Unuscione] [.marvel-2014.07.19][0] failed to start shard
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException:
[.marvel-2014.07.19][0] failed to recover shard
at
org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:241)
at
org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:132)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown
Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
Source)
at java.lang.Thread.run(Unknown Source)
Caused by: org.elasticsearch.ElasticsearchIllegalArgumentException: No
version type match [104]
at
org.elasticsearch.index.VersionType.fromValue(VersionType.java:307)
at
org.elasticsearch.index.translog.Translog$Create.readFrom(Translog.java:363)
at
org.elasticsearch.index.translog.TranslogStreams.readTranslogOperation(TranslogStreams.java:52)
at
org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:218)
... 4 more
[2014-07-22 13:08:46,126][WARN ][cluster.action.shard ] [Angelo
Unuscione] [.marvel-2014.07.20][0] sending failed shard for
[.marvel-2014.07.20][0], node[ifz3-S9fTgayreyuIXzHGA], [P],
s[INITIALIZING], indexUUID [B1PAAjUHQGilk8T8nQzTbQ], reason [Failed to
start shard, message
[IndexShardGatewayRecoveryException[[.marvel-2014.07.20][0] failed to
recover shard]; nested: ElasticsearchIllegalArgumentException[No version
type match [104]]; ]]
[

On Tuesday, July 22, 2014 10:29:53 AM UTC-4, Didjit wrote:

I've ran into some issues with Elasticsearch crashing on my windows box
(most likely my fault). Anyway, is there clean steps to brute force
recover? Something like, delete indexes, issue rebuild, wait .....

I do have the dev version of Marvel if there are tools in there.

Any help is appreciated. Thank you!

Chris

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/ba92816f-3cc1-48b2-b56b-2f2e299303aa%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/ba92816f-3cc1-48b2-b56b-2f2e299303aa%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEM624ZmqjiSrbcnKkbvNgKTxhUKCm3tPSFoc5W4FxWbxLfBBg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Didjit · July 23, 2014, 12:57am

How should I do that? I deleted the indexes through ES, then rebuilt
through ES. Is there a better way?

Tx

Chris

On Tuesday, July 22, 2014 6:17:27 PM UTC-4, Mark Walkom wrote:

It might be easier to just delete the data and reindex if you can (apart
from marvel).

There are a few lucene tools that you could run, but that's pretty
advanced, there isn't anything within ES itself.

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: ma...@campaignmonitor.com <javascript:>
web: www.campaignmonitor.com

On 23 July 2014 03:10, Didjit <didj...@gmail.com <javascript:>> wrote:

I keep getting these messages over and over. Any ideas how to repair?

Thank you.

[2014-07-22 13:08:43,966][WARN ][indices.cluster ] [Angelo
Unuscione] [.marvel-2014.07.19][0] failed to start shard
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException:
[.marvel-2014.07.19][0] failed to recover shard
at
org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:241)
at
org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:132)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown
Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
Source)
at java.lang.Thread.run(Unknown Source)
Caused by: org.elasticsearch.ElasticsearchIllegalArgumentException: No
version type match [104]
at
org.elasticsearch.index.VersionType.fromValue(VersionType.java:307)
at
org.elasticsearch.index.translog.Translog$Create.readFrom(Translog.java:363)
at
org.elasticsearch.index.translog.TranslogStreams.readTranslogOperation(TranslogStreams.java:52)
at
org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:218)
... 4 more
[2014-07-22 13:08:46,126][WARN ][cluster.action.shard ] [Angelo
Unuscione] [.marvel-2014.07.20][0] sending failed shard for
[.marvel-2014.07.20][0], node[ifz3-S9fTgayreyuIXzHGA], [P],
s[INITIALIZING], indexUUID [B1PAAjUHQGilk8T8nQzTbQ], reason [Failed to
start shard, message
[IndexShardGatewayRecoveryException[[.marvel-2014.07.20][0] failed to
recover shard]; nested: ElasticsearchIllegalArgumentException[No version
type match [104]]; ]]
[

On Tuesday, July 22, 2014 10:29:53 AM UTC-4, Didjit wrote:

I've ran into some issues with Elasticsearch crashing on my windows box
(most likely my fault). Anyway, is there clean steps to brute force
recover? Something like, delete indexes, issue rebuild, wait .....

I do have the dev version of Marvel if there are tools in there.

Any help is appreciated. Thank you!

Chris

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/ba92816f-3cc1-48b2-b56b-2f2e299303aa%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/ba92816f-3cc1-48b2-b56b-2f2e299303aa%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/b56dc5d3-3925-4f3e-9ebf-45e91552238f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Didjit · July 25, 2014, 12:57am

Bump nothing?

Everything was working fine for a few months and now I cant get much to
work. I went down to one node to reduce any conflicts between nodes.Deleted
and rebuilt indexes. Even reloaded data from the past, Issues I'm still
facing:

"This shows up all the time: [2014-07-24
20:50:36,927][DEBUG][action.search.type ] [Trip Monroe] All shards
failed for phase: [query]"

I have new indexes with new data, and Kibana does not recognize it (added
the correct index, created a dupe template (not named logstash).

Seems there should be some best practices to recover data out there. Lots
of pieces and parts, but I'm failing to find what will some my issues.
Almost ready to scrap it all and restart..... Ugh

Thanks

Chris

On Tuesday, July 22, 2014 8:57:12 PM UTC-4, Didjit wrote:

How should I do that? I deleted the indexes through ES, then rebuilt
through ES. Is there a better way?

Tx

Chris

On Tuesday, July 22, 2014 6:17:27 PM UTC-4, Mark Walkom wrote:

It might be easier to just delete the data and reindex if you can (apart
from marvel).

There are a few lucene tools that you could run, but that's pretty
advanced, there isn't anything within ES itself.

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: ma...@campaignmonitor.com
web: www.campaignmonitor.com

On 23 July 2014 03:10, Didjit didj...@gmail.com wrote:

I keep getting these messages over and over. Any ideas how to repair?

Thank you.

[2014-07-22 13:08:43,966][WARN ][indices.cluster ] [Angelo
Unuscione] [.marvel-2014.07.19][0] failed to start shard
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException:
[.marvel-2014.07.19][0] failed to recover shard
at
org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:241)
at
org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:132)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown
Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
Source)
at java.lang.Thread.run(Unknown Source)
Caused by: org.elasticsearch.ElasticsearchIllegalArgumentException: No
version type match [104]
at
org.elasticsearch.index.VersionType.fromValue(VersionType.java:307)
at
org.elasticsearch.index.translog.Translog$Create.readFrom(Translog.java:363)
at
org.elasticsearch.index.translog.TranslogStreams.readTranslogOperation(TranslogStreams.java:52)
at
org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:218)
... 4 more
[2014-07-22 13:08:46,126][WARN ][cluster.action.shard ] [Angelo
Unuscione] [.marvel-2014.07.20][0] sending failed shard for
[.marvel-2014.07.20][0], node[ifz3-S9fTgayreyuIXzHGA], [P],
s[INITIALIZING], indexUUID [B1PAAjUHQGilk8T8nQzTbQ], reason [Failed to
start shard, message
[IndexShardGatewayRecoveryException[[.marvel-2014.07.20][0] failed to
recover shard]; nested: ElasticsearchIllegalArgumentException[No version
type match [104]]; ]]
[

On Tuesday, July 22, 2014 10:29:53 AM UTC-4, Didjit wrote:

I've ran into some issues with Elasticsearch crashing on my windows box
(most likely my fault). Anyway, is there clean steps to brute force
recover? Something like, delete indexes, issue rebuild, wait .....

I do have the dev version of Marvel if there are tools in there.

Any help is appreciated. Thank you!

Chris

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/ba92816f-3cc1-48b2-b56b-2f2e299303aa%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/ba92816f-3cc1-48b2-b56b-2f2e299303aa%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/0d86c2f1-5976-4a16-a076-070f6d2d754b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.