# Help! After upgrading to Elasticsearch 6 cluster shard replicas will not allocate **URL:** \ **Category:** Elasticsearch\ **Created:** [November 17, 2017, 7:20am UTC](https://discuss.elastic.co/t/help-after-upgrading-to-elasticsearch-6-cluster-shard-replicas-will-not-allocate/108077 "2017-11-17T07:20:25Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![Warren\_Turner](https://avatars.discourse-cdn.com/v4/letter/w/e9c0ed/32.png) [@Warren\_Turner](https://discuss.elastic.co/u/Warren_Turner)\ **Post date:** [November 17, 2017, 7:20am UTC](https://discuss.elastic.co/t/help-after-upgrading-to-elasticsearch-6-cluster-shard-replicas-will-not-allocate/108077/1 "2017-11-17T07:20:25Z")
I upgraded my cluster to Elasticsearch 6 and while it is working, none of the redundant shards are being allocated ☹ When I use the explain API I get this: ``` curl -XGET 'localhost:9200/_cluster/allocation/explain?pretty' { "index" : "px-ext-access-2016.09.01", "shard" : 4, "primary" : false, "current_state" : "unassigned", "unassigned_info" : { "reason" : "REPLICA_ADDED", "at" : "2017-11-17T07:02:48.361Z", "last_allocation_status" : "no_attempt" }, "can_allocate" : "no", "allocate_explanation" : "cannot allocate because allocation is not permitted to any of the nodes", "node_allocation_decisions" : [ { "node_id" : "8SjCHI-OQn-mBAl0OBLx6Q", "node_name" : "bos1-es2", "transport_address" : "192.168.8.151:9300", "node_attributes" : { "ml.max_open_jobs" : "10", "ml.enabled" : "true" }, "node_decision" : "no", "deciders" : [ { "decider" : "enable", "decision" : "NO", "explanation" : "no allocations are allowed due to {}" } ] }, { "node_id" : "JR9qqMjEQF6c22eGZqIAcw", "node_name" : "bos1-es3", "transport_address" : "192.168.8.152:9300", "node_attributes" : { "ml.max_open_jobs" : "10", "ml.enabled" : "true" }, "node_decision" : "no", "deciders" : [ { "decider" : "enable", "decision" : "NO", "explanation" : "no allocations are allowed due to {}" }, { "decider" : "same_shard", "decision" : "NO", "explanation" : "the shard cannot be allocated to the same node on which a copy of the shard already exists [[px-ext-access-2016.09.01][4], node[JR9qqMjEQF6c22eGZqIAcw], [P], s[STARTED], a[id=IO9Hx2mpScqfigUPaX7mkQ]]" } ] }, { "node_id" : "tBZdi1W_TRa16getb8eNlA", "node_name" : "bos1-es1", "transport_address" : "192.168.8.150:9300", "node_attributes" : { "ml.max_open_jobs" : "10", "ml.enabled" : "true" }, "node_decision" : "no", "deciders" : [ { "decider" : "enable", "decision" : "NO", "explanation" : "no allocations are allowed due to {}" } ] } ] } ``` Help what do I do! I checked and all the cluster members are running 6.0.0 so I don't think there is a version mismatch issue. ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [November 17, 2017, 7:26am UTC](https://discuss.elastic.co/t/help-after-upgrading-to-elasticsearch-6-cluster-shard-replicas-will-not-allocate/108077/2 "2017-11-17T07:26:02Z")
Did you disable allocation when you started the upgrade? If so did you re-enable it? ---
**Author:** ![Warren\_Turner](https://avatars.discourse-cdn.com/v4/letter/w/e9c0ed/32.png) [@Warren\_Turner](https://discuss.elastic.co/u/Warren_Turner)\ **Post date:** [November 17, 2017, 7:50am UTC](https://discuss.elastic.co/t/help-after-upgrading-to-elasticsearch-6-cluster-shard-replicas-will-not-allocate/108077/3 "2017-11-17T07:50:32Z")
I figured it out just a minute ago. I had disabled allocation and re-enabled it but it didn't do anything. I tried "disabling" it and the re-enabling it again and it started correctly. Hurray for turning it off and on again 😃 ---
**Author:** ![Warren\_Turner](https://avatars.discourse-cdn.com/v4/letter/w/e9c0ed/32.png) [@Warren\_Turner](https://discuss.elastic.co/u/Warren_Turner)\ **Post date:** [November 17, 2017, 4:11pm UTC](https://discuss.elastic.co/t/help-after-upgrading-to-elasticsearch-6-cluster-shard-replicas-will-not-allocate/108077/4 "2017-11-17T16:11:34Z")
Uhoh, It looks like there are a few shards that still aren't getting allocated. I get this error: ``` curl -XGET 'localhost:9200/_cluster/allocation/explain?pretty' { "index" : "px-web-server-2017.11.17", "shard" : 4, "primary" : false, "current_state" : "unassigned", "unassigned_info" : { "reason" : "ALLOCATION_FAILED", "at" : "2017-11-17T07:27:37.708Z", "failed_allocation_attempts" : 5, "details" : "failed recovery, failure RecoveryFailedException[[px-web-server-2017.11.17][4]: Recovery failed from {bos1-es3}{JR9qqMjEQF6c22eGZqIAcw}{izoNnZpWTsyt5_xxymVDbg}{192.168.8.152}{192.168.8.152:9300}{ml.max_open_jobs=10, ml.enabled=true} into {bos1-es1}{tBZdi1W_TRa16getb8eNlA}{picaUSZyS7K34rTVpJvu1Q}{192.168.8.150}{192.168.8.150:9300}{ml.max_open_jobs=10, ml.enabled=true}]; nested: RemoteTransportException[[bos1-es3][192.168.8.152:9300][internal:index/shard/recovery/start_recovery]]; nested: RecoveryEngineException[Phase[2] phase2 failed]; nested: RemoteTransportException[[bos1-es1][192.168.8.150:9300][internal:index/shard/recovery/translog_ops]]; nested: TranslogException[Failed to write operation [Index{id='AV_HS9TBPU9Vhf7shs_U', type='px-web-server', seqNo=-2, primaryTerm=0}]]; nested: IllegalArgumentException[sequence number must be assigned]; ", "last_allocation_status" : "no_attempt" }, "can_allocate" : "no", "allocate_explanation" : "cannot allocate because allocation is not permitted to any of the nodes", "node_allocation_decisions" : [ { "node_id" : "8SjCHI-OQn-mBAl0OBLx6Q", "node_name" : "bos1-es2", "transport_address" : "192.168.8.151:9300", "node_attributes" : { "ml.max_open_jobs" : "10", "ml.enabled" : "true" }, "node_decision" : "no", "store" : { "matching_size_in_bytes" : 0 }, "deciders" : [ { "decider" : "max_retry", "decision" : "NO", "explanation" : "shard has exceeded the maximum number of retries [5] on failed allocation attempts - manually call [/_cluster/reroute?retry_failed=true] to retry, [unassigned_info[[reason=ALLOCATION_FAILED], at[2017-11-17T07:27:37.708Z], failed_attempts[5], delayed=false, details[failed recovery, failure RecoveryFailedException[[px-web-server-2017.11.17][4]: Recovery failed from {bos1-es3}{JR9qqMjEQF6c22eGZqIAcw}{izoNnZpWTsyt5_xxymVDbg}{192.168.8.152}{192.168.8.152:9300}{ml.max_open_jobs=10, ml.enabled=true} into {bos1-es1}{tBZdi1W_TRa16getb8eNlA}{picaUSZyS7K34rTVpJvu1Q}{192.168.8.150}{192.168.8.150:9300}{ml.max_open_jobs=10, ml.enabled=true}]; nested: RemoteTransportException[[bos1-es3][192.168.8.152:9300][internal:index/shard/recovery/start_recovery]]; nested: RecoveryEngineException[Phase[2] phase2 failed]; nested: RemoteTransportException[[bos1-es1][192.168.8.150:9300][internal:index/shard/recovery/translog_ops]]; nested: TranslogException[Failed to write operation [Index{id='AV_HS9TBPU9Vhf7shs_U', type='px-web-server', seqNo=-2, primaryTerm=0}]]; nested: IllegalArgumentException[sequence number must be assigned]; ], allocation_status[no_attempt]]]" } ] }, ``` ---
**Author:** ![Warren\_Turner](https://avatars.discourse-cdn.com/v4/letter/w/e9c0ed/32.png) [@Warren\_Turner](https://discuss.elastic.co/u/Warren_Turner)\ **Post date:** [November 17, 2017, 4:18pm UTC](https://discuss.elastic.co/t/help-after-upgrading-to-elasticsearch-6-cluster-shard-replicas-will-not-allocate/108077/5 "2017-11-17T16:18:30Z")
I tried running the retry command that the error suggests but I get: ``` "px-web-server-2017.11.17" : { "shards" : { "2" : [ { "state" : "STARTED", "primary" : true, "node" : "JR9qqMjEQF6c22eGZqIAcw", "relocating_node" : null, "shard" : 2, "index" : "px-web-server-2017.11.17", "allocation_id" : { "id" : "9X4ESJ3CS1OOnE3Y6V-KPA" } }, { "state" : "UNASSIGNED", "primary" : false, "node" : null, "relocating_node" : null, "shard" : 2, "index" : "px-web-server-2017.11.17", "recovery_source" : { "type" : "PEER" }, "unassigned_info" : { "reason" : "ALLOCATION_FAILED", "at" : "2017-11-17T16:15:11.973Z", "failed_attempts" : 6, "delayed" : false, "details" : "failed recovery, failure RecoveryFailedException[[px-web-server-2017.11.17][2]: Recovery failed from {bos1-es3}{JR9qqMjEQF6c22eGZqIAcw}{izoNnZpWTsyt5_xxymVDbg}{192.\ 168.8.152}{192.168.8.152:9300}{ml.max_open_jobs=10, ml.enabled=true} into {bos1-es1}{tBZdi1W_TRa16getb8eNlA}{picaUSZyS7K34rTVpJvu1Q}{192.168.8.150}{192.168.8.150:9300}{ml.max_open_jobs=10, ml.ena\ bled=true}]; nested: RemoteTransportException[[bos1-es3][192.168.8.152:9300][internal:index/shard/recovery/start_recovery]]; nested: RecoveryEngineException[Phase[2] phase2 failed]; nested: Remot\ eTransportException[[bos1-es1][192.168.8.150:9300][internal:index/shard/recovery/translog_ops]]; nested: TranslogException[Failed to write operation [Index{id='AV_HSw2kXITtF5uOUXel', type='px-web\ -server', seqNo=-2, primaryTerm=0}]]; nested: IllegalArgumentException[sequence number must be assigned]; ", "allocation_status" : "no_attempt" } } ], ``` ---
**Author:** ![Warren\_Turner](https://avatars.discourse-cdn.com/v4/letter/w/e9c0ed/32.png) [@Warren\_Turner](https://discuss.elastic.co/u/Warren_Turner)\ **Post date:** [November 17, 2017, 5:25pm UTC](https://discuss.elastic.co/t/help-after-upgrading-to-elasticsearch-6-cluster-shard-replicas-will-not-allocate/108077/6 "2017-11-17T17:25:45Z")
I think I'm just going to call it quits with this issue. It is only one index and it was being created during the rolling restart of the cluster so I bet it is in some screwy state. I'll just reindex it and not worry about it anymore 😉 ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [November 17, 2017, 8:02pm UTC](https://discuss.elastic.co/t/help-after-upgrading-to-elasticsearch-6-cluster-shard-replicas-will-not-allocate/108077/7 "2017-11-17T20:02:03Z")
If it's a replica, just set the replica count to 0 on the index, then add it back. ---
**Author:** ![Warren\_Turner](https://avatars.discourse-cdn.com/v4/letter/w/e9c0ed/32.png) [@Warren\_Turner](https://discuss.elastic.co/u/Warren_Turner)\ **Post date:** [November 17, 2017, 9:52pm UTC](https://discuss.elastic.co/t/help-after-upgrading-to-elasticsearch-6-cluster-shard-replicas-will-not-allocate/108077/8 "2017-11-17T21:52:07Z")
Hey that worked! Thanks Mark! ---
**Author:** ![bleskes](https://avatars.discourse-cdn.com/v4/letter/b/71c47a/32.png) [@bleskes](https://discuss.elastic.co/u/bleskes)\ **Post date:** [November 27, 2017, 1:32pm UTC](https://discuss.elastic.co/t/help-after-upgrading-to-elasticsearch-6-cluster-shard-replicas-will-not-allocate/108077/9 "2017-11-27T13:32:10Z")
@Warren_Turner can you please confirm that you did a rolling upgrade? Please see [https://github.com/elastic/elasticsearch/issues/27536](https://github.com/elastic/elasticsearch/issues/27536) for a description of why we think this happens. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [December 25, 2017, 1:32pm UTC](https://discuss.elastic.co/t/help-after-upgrading-to-elasticsearch-6-cluster-shard-replicas-will-not-allocate/108077/10 "2017-12-25T13:32:26Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.