# Upgrade gone bad **URL:** \ **Category:** Elasticsearch\ **Created:** [November 1, 2016, 2:27pm UTC](https://discuss.elastic.co/t/upgrade-gone-bad/64568 "2016-11-01T14:27:14Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![bigfootxxl](https://avatars.discourse-cdn.com/v4/letter/b/4af34b/32.png) [@bigfootxxl](https://discuss.elastic.co/u/bigfootxxl)\ **Post date:** [November 1, 2016, 2:27pm UTC](https://discuss.elastic.co/t/upgrade-gone-bad/64568/1 "2016-11-01T14:27:14Z")
I upgraded from 1.5.2 to 2.4.1 and now cluster is red and I have this looping in debug log between three shards it seems. One of them is this: [2016-11-01 16:26:09,264][WARN][cluster.action.shard] [company] [logstash-2015.05.06][3] received shard failed for target shard [[logstash-2015.05.06][3], node[fbFMhKcSQUSPVdKv40FxfA], [P], v[3484], s[INITIALIZING], a[id=1JoG3SCMQcCQZWX\_j-KsUQ], unassigned\_info[[reason=ALLOCATION\_FAILED], at[2016-11-01T14:26:08.684Z], details[failed recovery, failure IndexShardRecoveryException[failed recovery]; nested: IllegalStateException[translog.ckp file already present, translog is already upgraded]; ]]], indexUUID [YV93FwQTTk-pEFhsz2GgCg], message [failed recovery], failure [IndexShardRecoveryException[failed recovery]; nested: IllegalStateException[translog.ckp file already present, translog is already upgraded]; ] [logstash-2015.05.06][[logstash-2015.05.06][3]] IndexShardRecoveryException[failed recovery]; nested: IllegalStateException[translog.ckp file already present, translog is already upgraded]; at org.elasticsearch.index.shard.StoreRecoveryService$1.run(StoreRecoveryService.java:179) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.IllegalStateException: translog.ckp file already present, translog is already upgraded Can I fix it? ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [November 2, 2016, 8:46am UTC](https://discuss.elastic.co/t/upgrade-gone-bad/64568/2 "2016-11-02T08:46:22Z")
You can try moving the file it refers to aside. But that's no guarantee it'll work and recover. ---
**Author:** ![bigfootxxl](https://avatars.discourse-cdn.com/v4/letter/b/4af34b/32.png) [@bigfootxxl](https://discuss.elastic.co/u/bigfootxxl)\ **Post date:** [November 2, 2016, 9:44am UTC](https://discuss.elastic.co/t/upgrade-gone-bad/64568/3 "2016-11-02T09:44:16Z")
No that did not help: [2016-11-02 11:42:26,740][DEBUG][index.shard] [hostname] [logstash-2015.05.06][3] state: [CREATED]-\>[RECOVERING], reason [from store] [2016-11-02 11:42:26,752][DEBUG][index.shard] [hostname] [logstash-2015.05.06][3] updateBufferSize: engine is closed; skipping [2016-11-02 11:42:26,756][DEBUG][index.shard] [hostname] [logstash-2015.05.06][3] starting recovery from shard\_store ... [2016-11-02 11:42:26,764][DEBUG][index.shard] [hostname] [logstash-2015.05.06][3] updateBufferSize: engine is closed; skipping [2016-11-02 11:42:26,769][DEBUG][index.engine] [hostname] [logstash-2015.05.06][3] upgrading translog - no checkpoint found [2016-11-02 11:42:26,783][DEBUG][index] [hostname] [logstash-2015.05.06] [3] closing... (reason: [failed recovery]) [2016-11-02 11:42:26,783][DEBUG][index.shard] [hostname] [logstash-2015.05.06][3] state: [RECOVERING]-\>[CLOSED], reason [failed recovery] [2016-11-02 11:42:26,783][DEBUG][index.shard] [hostname] [logstash-2015.05.06][3] operations counter reached 0, will not accept any further writes [2016-11-02 11:42:26,783][DEBUG][index.store] [hostname] [logstash-2015.05.06][3] store reference count on close: 0 [2016-11-02 11:42:26,783][DEBUG][index] [hostname] [logstash-2015.05.06] [3] closed (reason: [failed recovery]) [2016-11-02 11:42:26,783][WARN][indices.cluster] [hostname] [[logstash-2015.05.06][3]] marking and sending shard failed due to [failed recovery] [logstash-2015.05.06][[logstash-2015.05.06][3]] IndexShardRecoveryException[failed recovery]; nested: IllegalStateException[non-legacy translog file [translog-1430859600187.tlog] found on a translog that wasn't upgraded yet]; [2016-11-02 11:42:26,783][WARN][cluster.action.shard] [hostname] [logstash-2015.05.06][3] received shard failed for target shard [[logstash-2015.05.06][3], node[fbFMhKcSQUSPVdKv40FxfA], [P], v[61485], s[INITIALIZING], a[id=G8c6D02LS8u4VCpajK7o7Q], unassigned\_info[[reason=ALLOCATION\_FAILED], at[2016-11-02T09:42:25.259Z], details[failed recovery, failure IndexShardRecoveryException[failed recovery]; nested: IllegalStateException[non-legacy translog file [translog-1430859600187.tlog] found on a translog that wasn't upgraded yet]; ]]], indexUUID [YV93FwQTTk-pEFhsz2GgCg], message [failed recovery], failure [IndexShardRecoveryException[failed recovery]; nested: IllegalStateException[non-legacy translog file [translog-1430859600187.tlog] found on a translog that wasn't upgraded yet]; ] [logstash-2015.05.06][[logstash-2015.05.06][3]] IndexShardRecoveryException[failed recovery]; nested: IllegalStateException[non-legacy translog file [translog-1430859600187.tlog] found on a translog that wasn't upgraded yet]; [2016-11-02 11:42:26,822][DEBUG][cluster.service] [hostname] processing [shard-failed ([logstash-2015.05.04][2], node[fbFMhKcSQUSPVdKv40FxfA], [P], v[61485], s[INITIALIZING], a[id=S2rKXaqIS9Ogiepeb3Ui3A], unassigned\_info[[reason=ALLOCATION\_FAILED], at[2016-11-02T09:42:25.259Z], details[failed recovery, failure IndexShardRecoveryException[failed recovery]; nested: IllegalStateException[non-legacy translog file [translog-1430686800112.tlog] found on a translog that wasn't upgraded yet]; ]]), message [failed recovery],shard-failed ([logstash-2015.05.06][3], node[fbFMhKcSQUSPVdKv40FxfA], [P], v[61485], s[INITIALIZING], a[id=G8c6D02LS8u4VCpajK7o7Q], unassigned\_info[[reason=ALLOCATION\_FAILED], at[2016-11-02T09:42:25.259Z], details[failed recovery, failure IndexShardRecoveryException[failed recovery]; nested: IllegalStateException[non-legacy translog file [translog-1430859600187.tlog] found on a translog that wasn't upgraded yet]; ]]), message [failed recovery]]: execute [2016-11-02 11:42:26,829][DEBUG][cluster.routing.allocation] [hostname] [logstash-2015.05.06][3] failed shard [logstash-2015.05.06][3], node[fbFMhKcSQUSPVdKv40FxfA], [P], v[61485], s[INITIALIZING], a[id=G8c6D02LS8u4VCpajK7o7Q], unassigned\_info[[reason=ALLOCATION\_FAILED], at[2016-11-02T09:42:25.259Z], details[failed recovery, failure IndexShardRecoveryException[failed recovery]; nested: IllegalStateException[non-legacy translog file [translog-1430859600187.tlog] found on a translog that wasn't upgraded yet]; ]] found in routingNodes, failing it ([reason=ALLOCATION\_FAILED], at[2016-11-02T09:42:26.829Z], details[failed recovery, failure IndexShardRecoveryException[failed recovery]; nested: IllegalStateException[non-legacy translog file [translog-1430859600187.tlog] found on a translog that wasn't upgraded yet]; ]) [2016-11-02 11:42:26,927][DEBUG][gateway] [hostname] [logstash-2015.05.06][3] loaded data path [/var/lib/elasticsearch/elasticsearch/nodes/0/indices/logstash-2015.05.06/3], state path [/var/lib/elasticsearch/elasticsearch/nodes/0/indices/logstash-2015.05.06/3] ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [November 2, 2016, 9:46am UTC](https://discuss.elastic.co/t/upgrade-gone-bad/64568/4 "2016-11-02T09:46:33Z")
How important is the data in the index? ---
**Author:** ![bigfootxxl](https://avatars.discourse-cdn.com/v4/letter/b/4af34b/32.png) [@bigfootxxl](https://discuss.elastic.co/u/bigfootxxl)\ **Post date:** [November 2, 2016, 10:02am UTC](https://discuss.elastic.co/t/upgrade-gone-bad/64568/5 "2016-11-02T10:02:44Z")
well... I can afford loosing that piece ([logstash-2015.05.06][3]). would be nice to retain the other 4 out of 5 that are ok. is that possible? ---
**Author:** ![bigfootxxl](https://avatars.discourse-cdn.com/v4/letter/b/4af34b/32.png) [@bigfootxxl](https://discuss.elastic.co/u/bigfootxxl)\ **Post date:** [November 2, 2016, 2:59pm UTC](https://discuss.elastic.co/t/upgrade-gone-bad/64568/6 "2016-11-02T14:59:07Z")
For everybody who reads this - in the end the shards were not corrupt, with kopf I was able to see that they were unassigned and assigning them to my only server fixed everything! ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 5, 2017, 10:07pm UTC](https://discuss.elastic.co/t/upgrade-gone-bad/64568/7 "2017-07-05T22:07:21Z")