# How to Cleanly Recover After Crash **URL:** \ **Category:** Elasticsearch\ **Created:** [July 22, 2014, 2:29pm UTC](https://discuss.elastic.co/t/how-to-cleanly-recover-after-crash/18794 "2014-07-22T14:29:52Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![Didjit](https://avatars.discourse-cdn.com/v4/letter/d/58f4c7/32.png) [@Didjit](https://discuss.elastic.co/u/Didjit)\ **Post date:** [July 22, 2014, 2:29pm UTC](https://discuss.elastic.co/t/how-to-cleanly-recover-after-crash/18794/1 "2014-07-22T14:29:52Z")
I've ran into some issues with Elasticsearch crashing on my windows box (most likely my fault). Anyway, is there clean steps to brute force recover? Something like, delete indexes, issue rebuild, wait ..... I do have the dev version of Marvel if there are tools in there. Any help is appreciated. Thank you! Chris -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/a45fff50-7c85-494f-a5ad-5c93892fa40f%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/a45fff50-7c85-494f-a5ad-5c93892fa40f%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Didjit](https://avatars.discourse-cdn.com/v4/letter/d/58f4c7/32.png) [@Didjit](https://discuss.elastic.co/u/Didjit)\ **Post date:** [July 22, 2014, 5:10pm UTC](https://discuss.elastic.co/t/how-to-cleanly-recover-after-crash/18794/2 "2014-07-22T17:10:01Z")
I keep getting these messages over and over. Any ideas how to repair? Thank you. * * * [2014-07-22 13:08:43,966][WARN][indices.cluster] [Angelo Unuscione] [.marvel-2014.07.19][0] failed to start shard org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [.marvel-2014.07.19][0] failed to recover shard at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:241) at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:132) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Caused by: org.elasticsearch.ElasticsearchIllegalArgumentException: No version type match [104] at org.elasticsearch.index.VersionType.fromValue(VersionType.java:307) at org.elasticsearch.index.translog.Translog$Create.readFrom(Translog.java:363) at org.elasticsearch.index.translog.TranslogStreams.readTranslogOperation(TranslogStreams.java:52) at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:218) ... 4 more [2014-07-22 13:08:46,126][WARN][cluster.action.shard] [Angelo Unuscione] [.marvel-2014.07.20][0] sending failed shard for [.marvel-2014.07.20][0], node[ifz3-S9fTgayreyuIXzHGA], [P], s[INITIALIZING], indexUUID [B1PAAjUHQGilk8T8nQzTbQ], reason [Failed to start shard, message [IndexShardGatewayRecoveryException[[.marvel-2014.07.20][0] failed to recover shard]; nested: ElasticsearchIllegalArgumentException[No version type match [104]]; ]] [ On Tuesday, July 22, 2014 10:29:53 AM UTC-4, Didjit wrote: > I've ran into some issues with Elasticsearch crashing on my windows box > (most likely my fault). Anyway, is there clean steps to brute force > recover? Something like, delete indexes, issue rebuild, wait ..... > > I do have the dev version of Marvel if there are tools in there. > > Any help is appreciated. Thank you! > > Chris -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/ba92816f-3cc1-48b2-b56b-2f2e299303aa%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/ba92816f-3cc1-48b2-b56b-2f2e299303aa%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [July 22, 2014, 10:16pm UTC](https://discuss.elastic.co/t/how-to-cleanly-recover-after-crash/18794/3 "2014-07-22T22:16:53Z")
It might be easier to just delete the data and reindex if you can (apart from marvel). There are a few lucene tools that you could run, but that's pretty advanced, there isn't anything within ES itself. Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: [markw@campaignmonitor.com](mailto:markw@campaignmonitor.com) web: [www.campaignmonitor.com](http://www.campaignmonitor.com) On 23 July 2014 03:10, Didjit [didjit86@gmail.com](mailto:didjit86@gmail.com) wrote: > I keep getting these messages over and over. Any ideas how to repair? > > Thank you. > > * * * > > [2014-07-22 13:08:43,966][WARN][indices.cluster] [Angelo > Unuscione] [.marvel-2014.07.19][0] failed to start shard > org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: > [.marvel-2014.07.19][0] failed to recover shard > at > org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:241) > at > org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:132) > at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown > Source) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown > Source) > at java.lang.Thread.run(Unknown Source) > Caused by: org.elasticsearch.ElasticsearchIllegalArgumentException: No > version type match [104] > at > org.elasticsearch.index.VersionType.fromValue(VersionType.java:307) > at > org.elasticsearch.index.translog.Translog$Create.readFrom(Translog.java:363) > at > org.elasticsearch.index.translog.TranslogStreams.readTranslogOperation(TranslogStreams.java:52) > at > org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:218) > ... 4 more > [2014-07-22 13:08:46,126][WARN][cluster.action.shard] [Angelo > Unuscione] [.marvel-2014.07.20][0] sending failed shard for > [.marvel-2014.07.20][0], node[ifz3-S9fTgayreyuIXzHGA], [P], > s[INITIALIZING], indexUUID [B1PAAjUHQGilk8T8nQzTbQ], reason [Failed to > start shard, message > [IndexShardGatewayRecoveryException[[.marvel-2014.07.20][0] failed to > recover shard]; nested: ElasticsearchIllegalArgumentException[No version > type match [104]]; ]] > [ > > On Tuesday, July 22, 2014 10:29:53 AM UTC-4, Didjit wrote: > > > I've ran into some issues with Elasticsearch crashing on my windows box > > (most likely my fault). Anyway, is there clean steps to brute force > > recover? Something like, delete indexes, issue rebuild, wait ..... > > > > I do have the dev version of Marvel if there are tools in there. > > > > Any help is appreciated. Thank you! > > > > Chris > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/ba92816f-3cc1-48b2-b56b-2f2e299303aa%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/ba92816f-3cc1-48b2-b56b-2f2e299303aa%40googlegroups.com) > [https://groups.google.com/d/msgid/elasticsearch/ba92816f-3cc1-48b2-b56b-2f2e299303aa%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/ba92816f-3cc1-48b2-b56b-2f2e299303aa%40googlegroups.com?utm_medium=email&utm_source=footer) > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAEM624ZmqjiSrbcnKkbvNgKTxhUKCm3tPSFoc5W4FxWbxLfBBg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEM624ZmqjiSrbcnKkbvNgKTxhUKCm3tPSFoc5W4FxWbxLfBBg%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Didjit](https://avatars.discourse-cdn.com/v4/letter/d/58f4c7/32.png) [@Didjit](https://discuss.elastic.co/u/Didjit)\ **Post date:** [July 23, 2014, 12:57am UTC](https://discuss.elastic.co/t/how-to-cleanly-recover-after-crash/18794/4 "2014-07-23T00:57:11Z")
How should I do that? I deleted the indexes through ES, then rebuilt through ES. Is there a better way? Tx Chris On Tuesday, July 22, 2014 6:17:27 PM UTC-4, Mark Walkom wrote: > It might be easier to just delete the data and reindex if you can (apart > from marvel). > > There are a few lucene tools that you could run, but that's pretty > advanced, there isn't anything within ES itself. > > Regards, > Mark Walkom > > Infrastructure Engineer > Campaign Monitor > email: [ma...@campaignmonitor.com](mailto:ma...@campaignmonitor.com) \ > web: [www.campaignmonitor.com](http://www.campaignmonitor.com) > > On 23 July 2014 03:10, Didjit \<[didj...@gmail.com](mailto:didj...@gmail.com) \\> wrote: > > > I keep getting these messages over and over. Any ideas how to repair? > > > > Thank you. > > > > * * * > > > > [2014-07-22 13:08:43,966][WARN][indices.cluster] [Angelo > > Unuscione] [.marvel-2014.07.19][0] failed to start shard > > org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: > > [.marvel-2014.07.19][0] failed to recover shard > > at > > org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:241) > > at > > org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:132) > > at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown > > Source) > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown > > Source) > > at java.lang.Thread.run(Unknown Source) > > Caused by: org.elasticsearch.ElasticsearchIllegalArgumentException: No > > version type match [104] > > at > > org.elasticsearch.index.VersionType.fromValue(VersionType.java:307) > > at > > org.elasticsearch.index.translog.Translog$Create.readFrom(Translog.java:363) > > at > > org.elasticsearch.index.translog.TranslogStreams.readTranslogOperation(TranslogStreams.java:52) > > at > > org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:218) > > ... 4 more > > [2014-07-22 13:08:46,126][WARN][cluster.action.shard] [Angelo > > Unuscione] [.marvel-2014.07.20][0] sending failed shard for > > [.marvel-2014.07.20][0], node[ifz3-S9fTgayreyuIXzHGA], [P], > > s[INITIALIZING], indexUUID [B1PAAjUHQGilk8T8nQzTbQ], reason [Failed to > > start shard, message > > [IndexShardGatewayRecoveryException[[.marvel-2014.07.20][0] failed to > > recover shard]; nested: ElasticsearchIllegalArgumentException[No version > > type match [104]]; ]] > > [ > > > > On Tuesday, July 22, 2014 10:29:53 AM UTC-4, Didjit wrote: > > > > > I've ran into some issues with Elasticsearch crashing on my windows box > > > (most likely my fault). Anyway, is there clean steps to brute force > > > recover? Something like, delete indexes, issue rebuild, wait ..... > > > > > > I do have the dev version of Marvel if there are tools in there. > > > > > > Any help is appreciated. Thank you! > > > > > > Chris > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/ba92816f-3cc1-48b2-b56b-2f2e299303aa%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/ba92816f-3cc1-48b2-b56b-2f2e299303aa%40googlegroups.com) > > [https://groups.google.com/d/msgid/elasticsearch/ba92816f-3cc1-48b2-b56b-2f2e299303aa%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/ba92816f-3cc1-48b2-b56b-2f2e299303aa%40googlegroups.com?utm_medium=email&utm_source=footer) > > . > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/b56dc5d3-3925-4f3e-9ebf-45e91552238f%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/b56dc5d3-3925-4f3e-9ebf-45e91552238f%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Didjit](https://avatars.discourse-cdn.com/v4/letter/d/58f4c7/32.png) [@Didjit](https://discuss.elastic.co/u/Didjit)\ **Post date:** [July 25, 2014, 12:57am UTC](https://discuss.elastic.co/t/how-to-cleanly-recover-after-crash/18794/5 "2014-07-25T00:57:13Z")
Bump nothing? Everything was working fine for a few months and now I cant get much to work. I went down to one node to reduce any conflicts between nodes.Deleted and rebuilt indexes. Even reloaded data from the past, Issues I'm still facing: "This shows up all the time: [2014-07-24 20:50:36,927][DEBUG][action.search.type] [Trip Monroe] All shards failed for phase: [query]" I have new indexes with new data, and Kibana does not recognize it (added the correct index, created a dupe template (not named logstash). Seems there should be some best practices to recover data out there. Lots of pieces and parts, but I'm failing to find what will some my issues. Almost ready to scrap it all and restart..... Ugh Thanks Chris On Tuesday, July 22, 2014 8:57:12 PM UTC-4, Didjit wrote: > How should I do that? I deleted the indexes through ES, then rebuilt > through ES. Is there a better way? > > Tx > > Chris > > On Tuesday, July 22, 2014 6:17:27 PM UTC-4, Mark Walkom wrote: > > > It might be easier to just delete the data and reindex if you can (apart > > from marvel). > > > > There are a few lucene tools that you could run, but that's pretty > > advanced, there isn't anything within ES itself. > > > > Regards, > > Mark Walkom > > > > Infrastructure Engineer > > Campaign Monitor > > email: [ma...@campaignmonitor.com](mailto:ma...@campaignmonitor.com) > > web: [www.campaignmonitor.com](http://www.campaignmonitor.com) > > > > On 23 July 2014 03:10, Didjit [didj...@gmail.com](mailto:didj...@gmail.com) wrote: > > > > > I keep getting these messages over and over. Any ideas how to repair? > > > > > > Thank you. > > > > > > * * * > > > > > > [2014-07-22 13:08:43,966][WARN][indices.cluster] [Angelo > > > Unuscione] [.marvel-2014.07.19][0] failed to start shard > > > org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: > > > [.marvel-2014.07.19][0] failed to recover shard > > > at > > > org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:241) > > > at > > > org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:132) > > > at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown > > > Source) > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown > > > Source) > > > at java.lang.Thread.run(Unknown Source) > > > Caused by: org.elasticsearch.ElasticsearchIllegalArgumentException: No > > > version type match [104] > > > at > > > org.elasticsearch.index.VersionType.fromValue(VersionType.java:307) > > > at > > > org.elasticsearch.index.translog.Translog$Create.readFrom(Translog.java:363) > > > at > > > org.elasticsearch.index.translog.TranslogStreams.readTranslogOperation(TranslogStreams.java:52) > > > at > > > org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:218) > > > ... 4 more > > > [2014-07-22 13:08:46,126][WARN][cluster.action.shard] [Angelo > > > Unuscione] [.marvel-2014.07.20][0] sending failed shard for > > > [.marvel-2014.07.20][0], node[ifz3-S9fTgayreyuIXzHGA], [P], > > > s[INITIALIZING], indexUUID [B1PAAjUHQGilk8T8nQzTbQ], reason [Failed to > > > start shard, message > > > [IndexShardGatewayRecoveryException[[.marvel-2014.07.20][0] failed to > > > recover shard]; nested: ElasticsearchIllegalArgumentException[No version > > > type match [104]]; ]] > > > [ > > > > > > On Tuesday, July 22, 2014 10:29:53 AM UTC-4, Didjit wrote: > > > > > > > I've ran into some issues with Elasticsearch crashing on my windows box > > > > (most likely my fault). Anyway, is there clean steps to brute force > > > > recover? Something like, delete indexes, issue rebuild, wait ..... > > > > > > > > I do have the dev version of Marvel if there are tools in there. > > > > > > > > Any help is appreciated. Thank you! > > > > > > > > Chris > > > > > > -- > > > You received this message because you are subscribed to the Google > > > Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > To view this discussion on the web visit > > > [https://groups.google.com/d/msgid/elasticsearch/ba92816f-3cc1-48b2-b56b-2f2e299303aa%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/ba92816f-3cc1-48b2-b56b-2f2e299303aa%40googlegroups.com) > > > [https://groups.google.com/d/msgid/elasticsearch/ba92816f-3cc1-48b2-b56b-2f2e299303aa%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/ba92816f-3cc1-48b2-b56b-2f2e299303aa%40googlegroups.com?utm_medium=email&utm_source=footer) > > > . > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/0d86c2f1-5976-4a16-a076-070f6d2d754b%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/0d86c2f1-5976-4a16-a076-070f6d2d754b%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 1:13am UTC](https://discuss.elastic.co/t/how-to-cleanly-recover-after-crash/18794/6 "2017-07-06T01:13:15Z")