# ES upgrade shards issue - ES 5.6.10 to ES 6.3 **URL:** \ **Category:** Elasticsearch\ **Created:** [June 21, 2018, 2:42pm UTC](https://discuss.elastic.co/t/es-upgrade-shards-issue-es-5-6-10-to-es-6-3/136899 "2018-06-21T14:42:20Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![sureshvytla](https://avatars.discourse-cdn.com/v4/letter/s/e95f7d/32.png) [@sureshvytla](https://discuss.elastic.co/u/sureshvytla)\ **Post date:** [June 21, 2018, 2:42pm UTC](https://discuss.elastic.co/t/es-upgrade-shards-issue-es-5-6-10-to-es-6-3/136899/1 "2018-06-21T14:42:20Z")
Team, We have two node ES cluster which has been upgraded to 5.6.10 and we are having issue with shards in upgrading ES 5.6.10 to ES 6.3, in the both models either Rolling upgrade or full cluster restart we couldn't success. ``` [2018-06-20T06:06:00,590][WARN][o.e.c.r.a.AllocationService] [host01.xxx] failing shard [failed shard, shard [my-201708][2], node[s-PbDSILSmKRYlTO4_FOvw], [R], recovery_source[peer recovery], s[INITIALIZING], a[id=aqQb_CrCS-WmyRcrSFtF6w], unassigned_info[[reason=ALLOCATION_FAILED], at[2018-06-20T06:05:59.714Z], failed_attempts[4], delayed=false, details[failed shard on node [s-PbDSILSmKRYlTO4_FOvw]: failed recovery, failure RecoveryFailedException[[my-201708][2]: Recovery failed from {host01.xxx}{wsV326J5QxGdk31A7XY38w}{dhXjygn0Sg-tcKICCSRkiw}{host01.xxx}{150.0.1.242:9300}{ml.machine_memory=32898998272, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true} into {host02.xxx}{s-PbDSILSmKRYlTO4_FOvw}{VAJjb_PvTbe03nBZE_HBKA}{host02.xxx}{150.0.1.149:9300}{ml.machine_memory=32898998272, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true}]; nested: RemoteTransportException[[host01.xxx][150.0.1.242:9300][internal:index/shard/recovery/start_recovery]]; nested: RecoveryEngineException[Phase[1] prepare target for translog failed]; nested: RemoteTransportException[[host02.xxx][150.0.1.149:9300][internal:index/shard/recovery/prepare_translog]]; nested: IllegalStateException[commit doesn't contain history uuid]; ], allocation_status[no_attempt]], message [failed recovery], failure [RecoveryFailedException[[my-201708][2]: Recovery failed from {host01.xxx}{wsV326J5QxGdk31A7XY38w}{dhXjygn0Sg-tcKICCSRkiw}{host01.xxx}{150.0.1.242:9300}{ml.machine_memory=32898998272, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true} into {host02.xxx}{s-PbDSILSmKRYlTO4_FOvw}{VAJjb_PvTbe03nBZE_HBKA}{host02.xxx}{150.0.1.149:9300}{ml.machine_memory=32898998272, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true}]; nested: RemoteTransportException[[host01.xxx][150.0.1.242:9300][internal:index/shard/recovery/start_recovery]]; nested: RecoveryEngineException[Phase[1] prepare target for translog failed]; nested: RemoteTransportException[[host02.xxx][150.0.1.149:9300][internal:index/shard/recovery/prepare_translog]]; nested: IllegalStateException[commit doesn't contain history uuid]; ], markAsStale [true]] org.elasticsearch.indices.recovery.RecoveryFailedException: [my-201708][2]: Recovery failed from {host01.xxx}{wsV326J5QxGdk31A7XY38w}{dhXjygn0Sg-tcKICCSRkiw}{host01.xxx}{150.0.1.242:9300}{ml.machine_memory=32898998272, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true} into {host02.xxx}{s-PbDSILSmKRYlTO4_FOvw}{VAJjb_PvTbe03nBZE_HBKA}{host02.xxx}{150.0.1.149:9300}{ml.machine_memory=32898998272, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true} at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.doRecovery(PeerRecoveryTargetService.java:282) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.indices.recovery.PeerRecoveryTargetService.access$900(PeerRecoveryTargetService.java:80) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.indices.recovery.PeerRecoveryTargetService$RecoveryRunner.doRun(PeerRecoveryTargetService.java:623) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:724) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-6.3.0.jar:6.3.0] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_121] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_121] at java.lang.Thread.run(Thread.java:745) [?:1.8.0_121] Caused by: org.elasticsearch.transport.RemoteTransportException: [host01.xxx][150.0.1.242:9300][internal:index/shard/recovery/start_recovery] Caused by: org.elasticsearch.index.engine.RecoveryEngineException: Phase[1] prepare target for translog failed at org.elasticsearch.indices.recovery.RecoverySourceHandler.recoverToTarget(RecoverySourceHandler.java:191) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.indices.recovery.PeerRecoverySourceService.recover(PeerRecoverySourceService.java:98) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.indices.recovery.PeerRecoverySourceService.access$000(PeerRecoverySourceService.java:50) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.indices.recovery.PeerRecoverySourceService$StartRecoveryTransportRequestHandler.messageReceived(PeerRecoverySourceService.java:107) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.indices.recovery.PeerRecoverySourceService$StartRecoveryTransportRequestHandler.messageReceived(PeerRecoverySourceService.java:104) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.transport.TransportRequestHandler.messageReceived(TransportRequestHandler.java:30) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$1.doRun(SecurityServerTransportInterceptor.java:246) ~[?:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportInterceptor.java:304) ~[?:?] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:66) ~[elasticsearch-6.3.0.jar:6.3.0] at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1592) ~[elasticsearch-6.3.0.jar:6.3.0] ... 5 more ``` We would like to upgrade to ES 6 to resolve CVEs in lucene, please advice on this. Thanks, Suresh Vytla ---
**Author:** ![nhat](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nhat/32/22170_2.png) [@nhat](https://discuss.elastic.co/u/nhat)\ **Post date:** [June 22, 2018, 4:08am UTC](https://discuss.elastic.co/t/es-upgrade-shards-issue-es-5-6-10-to-es-6-3/136899/2 "2018-06-22T04:08:32Z")
@sureshvytla Sadly, this is a [known issue](https://github.com/elastic/elasticsearch/issues/31482). The [fix](https://github.com/elastic/elasticsearch/pull/31506) will be included in 6.3.1. A workaround, in this case, is to rebuild the replicas of the offending index (i.e. `my-201708`). This can be done by changing to the number of replicas to 0, then restoring it to the original value. Step 1: ```auto PUT /my-201708/_settings { "index" : { "number_of_replicas" : 0 } } ``` Step 2: ```auto PUT /my-201708/_settings { "index" : { "number_of_replicas" : 1 // the original value } } ``` Hope this helps. ---
**Author:** ![sureshvytla](https://avatars.discourse-cdn.com/v4/letter/s/e95f7d/32.png) [@sureshvytla](https://discuss.elastic.co/u/sureshvytla)\ **Post date:** [June 22, 2018, 3:57pm UTC](https://discuss.elastic.co/t/es-upgrade-shards-issue-es-5-6-10-to-es-6-3/136899/3 "2018-06-22T15:57:51Z")
@nhat - Thank you so much for the detailed explanation and we would like to go with 6.3.1 release, is there any ETA on ES 6.3.1 release. ---
**Author:** ![nhat](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nhat/32/22170_2.png) [@nhat](https://discuss.elastic.co/u/nhat)\ **Post date:** [June 22, 2018, 4:12pm UTC](https://discuss.elastic.co/t/es-upgrade-shards-issue-es-5-6-10-to-es-6-3/136899/4 "2018-06-22T16:12:42Z")
@sureshvytla We are working on the new release but the date/time is still unknown. ---
**Author:** ![nhat](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nhat/32/22170_2.png) [@nhat](https://discuss.elastic.co/u/nhat)\ **Post date:** [June 25, 2018, 2:52pm UTC](https://discuss.elastic.co/t/es-upgrade-shards-issue-es-5-6-10-to-es-6-3/136899/5 "2018-06-25T14:52:13Z")
For the future readers, @bleskes has a cleaner workaround for this issue. It consists of these two steps: - Step 1: Force flush the offending index `POST /my-201708/_flush?force=true` - Step 2: Retry the cluster allocation `POST /_cluster/reroute?retry_failed` ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 23, 2018, 2:52pm UTC](https://discuss.elastic.co/t/es-upgrade-shards-issue-es-5-6-10-to-es-6-3/136899/6 "2018-07-23T14:52:23Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.