# RecoveryFailedException while adding a node during Elasticsearch upgrade from 0.2.0 to 1.2.1 **URL:** \ **Category:** Elasticsearch\ **Created:** [June 24, 2014, 11:14pm UTC](https://discuss.elastic.co/t/recoveryfailedexception-while-adding-a-node-during-elasticsearch-upgrade-from-0-2-0-to-1-2-1/18310 "2014-06-24T23:14:58Z")\ **Posts on this page:** 3\ **Page:** 1
**Author:** ![anurag\_naidu](https://avatars.discourse-cdn.com/v4/letter/a/cdc98d/32.png) [@anurag\_naidu](https://discuss.elastic.co/u/anurag_naidu)\ **Post date:** [June 24, 2014, 11:14pm UTC](https://discuss.elastic.co/t/recoveryfailedexception-while-adding-a-node-during-elasticsearch-upgrade-from-0-2-0-to-1-2-1/18310/1 "2014-06-24T23:14:58Z")
We are upgrading our ES server from 0.2.0 to 1.2.1 using the cluster restart upgrade steps here [http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/setup-upgrade.html#restart-upgrade](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/setup-upgrade.html#restart-upgrade). We have 2 nodes as part of our cluster. We disabled allocation on both nodes before the upgrade and then upgraded the servers. However after starting both nodes, when we attempt to re-enable allocation it seem like Node2 just doesn't work with shard reallocation and throws the following error. The errors just keep happening so we eventually had to turn off replication again and now we are getting by with a single node in the cluster with no replication. Below is the stacktrace of the NullPointerException when trying to enable allocation. Any pointers to help with solving this would be greatly appreciated. I can provide more details if needed. [WARN][indices.cluster] [Valentina Allegra de La Fontaine] [production\_restaurants][2] failed to start shard org.elasticsearch.indices.recovery.RecoveryFailedException: [production\_restaurants][2]: Recovery failed from [Shadowmage][Ei4dGmkmScmY0WOPN8PR\_Q][server][inet[/server\_ip:9300]] int o [Valentina Allegra de La Fontaine][LJO\_jO59QGuGa-jLmTOZWg][server][inet[/server\_ip:9300]] at org.elasticsearch.indices.recovery.RecoveryTarget.doRecovery(RecoveryTarget.java:306) at org.elasticsearch.indices.recovery.RecoveryTarget.access$300(RecoveryTarget.java:65) at org.elasticsearch.indices.recovery.RecoveryTarget$2.run(RecoveryTarget.java:175) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744) Caused by: org.elasticsearch.transport.RemoteTransportException: [Shadowmage][inet[/172.16.21.21:9300]][index/shard/recovery/startRecovery] Caused by: org.elasticsearch.index.engine.RecoveryEngineException: [production\_restaurants][2] Phase[1] Execution failed at org.elasticsearch.index.engine.internal.InternalEngine.recover(InternalEngine.java:996) at org.elasticsearch.index.shard.service.InternalIndexShard.recover(InternalIndexShard.java:631) at org.elasticsearch.indices.recovery.RecoverySource.recover(RecoverySource.java:122) at org.elasticsearch.indices.recovery.RecoverySource.access$1600(RecoverySource.java:62) at org.elasticsearch.indices.recovery.RecoverySource$StartRecoveryTransportRequestHandler.messageReceived(RecoverySource.java:351) at org.elasticsearch.indices.recovery.RecoverySource$StartRecoveryTransportRequestHandler.messageReceived(RecoverySource.java:337) at org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:270) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744) Caused by: org.elasticsearch.indices.recovery.RecoverFilesRecoveryException: [production\_restaurants][2] Failed to transfer [44] files with total size of [904.2mb] at org.elasticsearch.indices.recovery.RecoverySource$1.phase1(RecoverySource.java:243) at org.elasticsearch.index.engine.internal.InternalEngine.recover(InternalEngine.java:993) ... 9 more Caused by: java.lang.NullPointerException Thanks -anurag -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/1788dbdf-7e0c-4f7d-8382-01ad47a15cb7%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/1788dbdf-7e0c-4f7d-8382-01ad47a15cb7%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\ **Post date:** [June 30, 2014, 7:20am UTC](https://discuss.elastic.co/t/recoveryfailedexception-while-adding-a-node-during-elasticsearch-upgrade-from-0-2-0-to-1-2-1/18310/2 "2014-06-30T07:20:13Z")
Hey, can you check your log files and have access to to the full stack trace of the NullPointerException in order to find out what is going on? Thanks! --Alex On Wed, Jun 25, 2014 at 1:14 AM, anurag naidu [anuragnaidu@gmail.com](mailto:anuragnaidu@gmail.com) wrote: > We are upgrading our ES server from 0.2.0 to 1.2.1 using the cluster > restart upgrade steps here > [http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/setup-upgrade.html#restart-upgrade](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/setup-upgrade.html#restart-upgrade). > We have 2 nodes as part of our cluster. We disabled allocation on both > nodes before the upgrade and then upgraded the servers. > > However after starting both nodes, when we attempt to re-enable allocation > it seem like Node2 just doesn't work with shard reallocation and throws the > following error. The errors just keep happening so we eventually had to > turn off replication again and now we are getting by with a single node in > the cluster with no replication. > > Below is the stacktrace of the NullPointerException when trying to enable > allocation. Any pointers to help with solving this would be greatly > appreciated. I can provide more details if needed. > > [WARN][indices.cluster] [Valentina Allegra de La Fontaine] > [production\_restaurants][2] failed to start shard > org.elasticsearch.indices.recovery.RecoveryFailedException: > [production\_restaurants][2]: Recovery failed from > [Shadowmage][Ei4dGmkmScmY0WOPN8PR\_Q][server][inet[/server\_ip:9300]] int > o [Valentina Allegra de La > Fontaine][LJO\_jO59QGuGa-jLmTOZWg][server][inet[/server\_ip:9300]] > at > org.elasticsearch.indices.recovery.RecoveryTarget.doRecovery(RecoveryTarget.java:306) > at > org.elasticsearch.indices.recovery.RecoveryTarget.access$300(RecoveryTarget.java:65) > at > org.elasticsearch.indices.recovery.RecoveryTarget$2.run(RecoveryTarget.java:175) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) > Caused by: org.elasticsearch.transport.RemoteTransportException: > [Shadowmage][inet[/172.16.21.21:9300]][index/shard/recovery/startRecovery] > Caused by: org.elasticsearch.index.engine.RecoveryEngineException: > [production\_restaurants][2] Phase[1] Execution failed > at > org.elasticsearch.index.engine.internal.InternalEngine.recover(InternalEngine.java:996) > at > org.elasticsearch.index.shard.service.InternalIndexShard.recover(InternalIndexShard.java:631) > at > org.elasticsearch.indices.recovery.RecoverySource.recover(RecoverySource.java:122) > at > org.elasticsearch.indices.recovery.RecoverySource.access$1600(RecoverySource.java:62) > at > org.elasticsearch.indices.recovery.RecoverySource$StartRecoveryTransportRequestHandler.messageReceived(RecoverySource.java:351) > at > org.elasticsearch.indices.recovery.RecoverySource$StartRecoveryTransportRequestHandler.messageReceived(RecoverySource.java:337) > at > org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:270) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) > Caused by: > org.elasticsearch.indices.recovery.RecoverFilesRecoveryException: > [production\_restaurants][2] Failed to transfer [44] files with total size > of [904.2mb] > at > org.elasticsearch.indices.recovery.RecoverySource$1.phase1(RecoverySource.java:243) > at > org.elasticsearch.index.engine.internal.InternalEngine.recover(InternalEngine.java:993) > ... 9 more > Caused by: java.lang.NullPointerException > > Thanks > -anurag > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/1788dbdf-7e0c-4f7d-8382-01ad47a15cb7%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/1788dbdf-7e0c-4f7d-8382-01ad47a15cb7%40googlegroups.com) > [https://groups.google.com/d/msgid/elasticsearch/1788dbdf-7e0c-4f7d-8382-01ad47a15cb7%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/1788dbdf-7e0c-4f7d-8382-01ad47a15cb7%40googlegroups.com?utm_medium=email&utm_source=footer) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAGCwEM81i8JFJ\_F5i%2BfFoKO-ExkSfwu-k-qHE2JaTjCBMYc9hQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAGCwEM81i8JFJ_F5i%2BfFoKO-ExkSfwu-k-qHE2JaTjCBMYc9hQ%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 1:19am UTC](https://discuss.elastic.co/t/recoveryfailedexception-while-adding-a-node-during-elasticsearch-upgrade-from-0-2-0-to-1-2-1/18310/3 "2017-07-06T01:19:00Z")