# Problems upgrading to 1.5.0 **URL:** \ **Category:** Elasticsearch\ **Created:** [March 31, 2015, 9:19am UTC](https://discuss.elastic.co/t/problems-upgrading-to-1-5-0/23000 "2015-03-31T09:19:36Z")\ **Posts on this page:** 2\ **Page:** 1
**Author:** ![maf](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/maf/32/77965_2.png) [@maf](https://discuss.elastic.co/u/maf)\ **Post date:** [March 31, 2015, 9:19am UTC](https://discuss.elastic.co/t/problems-upgrading-to-1-5-0/23000/1 "2015-03-31T09:19:36Z")
We just encountered some mysterious problems when upgrading from 1.1.1 to 1.5.0. The cluster consists of three machines, two data nodes and one master-only node. It hosts 86 indices which each has one replica. I stopped writes, did a snapshot and stopped the entire cluster before I upgraded the nodes and restarted them. The system came up and quickly turned yellow, but it refused to become green. it failed to recover a number of shards. The errors I got in the logs looked like this (there were a lot): [2015-03-31 07:33:39,704][WARN][indices.cluster] [NODE1] [signal\_bin][0] sending failed shard after recovery failure org.elasticsearch.indices.recovery.RecoveryFailedException: [signal\_bin][0]: Recovery failed from [NODE2][rpXLVgS8Qw2jgimXNYKn\_A][NODE2][inet[/IP2:9300]]{aws\_availability\_zone=us-east-1d, max\_local\_storage\_nodes=1} into [NODE1][tdXdf0MeS62DIO0KFZX-Rg][NODE1][inet[/IP1:9300]]{aws\_availability\_zone=us-east-1b, max\_local\_storage\_nodes=1} at org.elasticsearch.indices.recovery.RecoveryTarget.doRecovery(RecoveryTarget.java:274) at org.elasticsearch.indices.recovery.RecoveryTarget.access$700(RecoveryTarget.java:69) at org.elasticsearch.indices.recovery.RecoveryTarget$RecoveryRunner.doRun(RecoveryTarget.java:550) at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:36) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:724) Caused by: org.elasticsearch.transport.RemoteTransportException: [NODE2][inet[/IP2:9300]][internal:index/shard/recovery/start\_recovery] Caused by: org.elasticsearch.index.engine.RecoveryEngineException: [signal\_bin][0] Phase[1] Execution failed at org.elasticsearch.index.engine.InternalEngine.recover(InternalEngine.java:839) at org.elasticsearch.index.shard.IndexShard.recover(IndexShard.java:684) at org.elasticsearch.indices.recovery.RecoverySource.recover(RecoverySource.java:125) at org.elasticsearch.indices.recovery.RecoverySource.access$200(RecoverySource.java:49) at org.elasticsearch.indices.recovery.RecoverySource$StartRecoveryTransportRequestHandler.messageReceived(RecoverySource.java:146) at org.elasticsearch.indices.recovery.RecoverySource$StartRecoveryTransportRequestHandler.messageReceived(RecoverySource.java:132) at org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.doRun(MessageChannelHandler.java:279) at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:36) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:724) Caused by: org.elasticsearch.indices.recovery.RecoverFilesRecoveryException: [signal\_bin][0] Failed to transfer [11] files with total size of [1.4mb] at org.elasticsearch.indices.recovery.RecoverySourceHandler.phase1(RecoverySourceHandler.java:413) at org.elasticsearch.index.engine.InternalEngine.recover(InternalEngine.java:834) ... 10 more Caused by: org.elasticsearch.transport.RemoteTransportException: [NODE1][inet[/IP1:9300]][internal:index/shard/recovery/clean\_files] Caused by: org.elasticsearch.indices.recovery.RecoveryFailedException: [signal\_bin][0]: Recovery failed from [NODE2][rpXLVgS8Qw2jgimXNYKn\_A][NODE2][inet[/IP2:9300]]{aws\_availability\_zone=us-east-1d, max\_local\_storage\_nodes=1} into [NODE1][tdXdf0MeS62DIO0KFZX-Rg][NODE1][inet[/IP1:9300]]{aws\_availability\_zone=us-east-1b, max\_local\_storage\_nodes=1} (failed to clean after recovery) at org.elasticsearch.indices.recovery.RecoveryTarget$CleanFilesRequestHandler.messageReceived(RecoveryTarget.java:443) at org.elasticsearch.indices.recovery.RecoveryTarget$CleanFilesRequestHandler.messageReceived(RecoveryTarget.java:389) at org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.doRun(MessageChannelHandler.java:279) at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:36) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:724) Caused by: org.elasticsearch.ElasticsearchIllegalStateException: local version: name [\_yor.si], length [363], checksum [1jnqbzx], writtenBy [null] is different from remote version after recovery: name [\_yor.si], length [363], checksum [null], writtenBy [null] at org.elasticsearch.index.store.Store.verifyAfterCleanup(Store.java:645) at org.elasticsearch.index.store.Store.cleanupAndVerify(Store.java:613) at org.elasticsearch.indices.recovery.RecoveryTarget$CleanFilesRequestHandler.messageReceived(RecoveryTarget.java:428) ... 6 more The index/shard mentioned varied. We finally got past his by configuring the troubling indices to have 0 replicas and then back to 1. Has anybody seen something similar? Did we hit a bug or did we do something wrong? ``` /MaF ``` -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/f0fbd06f-0b08-49aa-a387-b78a081be59f%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/f0fbd06f-0b08-49aa-a387-b78a081be59f%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 12:22am UTC](https://discuss.elastic.co/t/problems-upgrading-to-1-5-0/23000/2 "2017-07-06T00:22:52Z")