# Problem with shards when upgrading to 2.3.3 **URL:** \ **Category:** Elasticsearch\ **Created:** [July 4, 2016, 11:14am UTC](https://discuss.elastic.co/t/problem-with-shards-when-upgrading-to-2-3-3/54639 "2016-07-04T11:14:14Z")\ **Posts on this page:** 14\ **Page:** 1
**Author:** ![misomij](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/misomij/32/10700_2.png) [@misomij](https://discuss.elastic.co/u/misomij)\ **Post date:** [July 4, 2016, 11:14am UTC](https://discuss.elastic.co/t/problem-with-shards-when-upgrading-to-2-3-3/54639/1 "2016-07-04T11:14:14Z")
Hi, I have upgraded a cluster of 3 nodes from 2.1 to 2.3.3, now i have problems with shards replicas not being allocated on all my nodes: ![](https://us1.discourse-cdn.com/elastic/original/2X/4/4da81b0fcdf0a65d36645d9d20aefe1912e109a9.png) I have upgraded the nodes this way: 1 turn off es 2 upgraded es with yum repositories 3 updated node with yum 4 checked configurations and permissions 5 started es When the first node started it joined the cluster and everything became green. After the second node has upgraded replicas won't allocate on that node. I thought that the problem could be the last non upgraded node which became master, so after a while I upgraded also it. While upgrading the replicas started allocating on the second node (the one on which they won't allocate before), then after the upgrade of the third node finished i had the same problem on this, replicas won't allocate. I decided to wait the we because i read that on 2.3.3 indexing can be very slow, now the state is what you can see in the image, new indices allocate on all nodes, old indices don't. Nodes are centos 7 with logstash 2.3.3 and jdk1.8 Has anyone idea on what causes this problem? Thank you, Miso ---
**Author:** ![ywelsch](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ywelsch/32/7751_2.png) [@ywelsch](https://discuss.elastic.co/u/ywelsch)\ **Post date:** [July 4, 2016, 12:16pm UTC](https://discuss.elastic.co/t/problem-with-shards-when-upgrading-to-2-3-3/54639/2 "2016-07-04T12:16:29Z")
Please try a reroute command with the explain parameter turned on: (see [https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-reroute.html](https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-reroute.html) ) ```auto curl -XPOST 'localhost:9200/_cluster/reroute?explain' -d '{ "commands" : [ { "allocate" : { "index" : "winlogbeat-2016.07.01", "shard" : 0, "node" : "ith-grs-sec-centos03" } } ] }' ``` ---
**Author:** ![misomij](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/misomij/32/10700_2.png) [@misomij](https://discuss.elastic.co/u/misomij)\ **Post date:** [July 4, 2016, 1:13pm UTC](https://discuss.elastic.co/t/problem-with-shards-when-upgrading-to-2-3-3/54639/3 "2016-07-04T13:13:25Z")
Hi, i launched the command, here's the output: > { > "acknowledged": true, > "explanations": [ > { > "command": "allocate", > "decisions": [ > { > "decider": "filter", > "decision": "YES", > "explanation": "node passes include/exclude/require filters" > }, > { > "decider": "enable", > "decision": "YES", > "explanation": "allocation disabling is ignored" > }, > { > "decider": "shards\_limit", > "decision": "YES", > "explanation": "total shard limit disabled: [index: -1, cluster: -1] \<= 0" > }, > { > "decider": "same\_shard", > "decision": "YES", > "explanation": "shard is not allocated to same node or host" > }, > { > "decider": "awareness", > "decision": "YES", > "explanation": "no allocation awareness enabled" > }, > { > "decider": "disable", > "decision": "YES", > "explanation": "allocation disabling is ignored" > }, > { > "decider": "node\_version", > "decision": "YES", > "explanation": "target node version [2.3.3] is same or newer than source node version [2.3.3]" > }, > { > "decider": "disk\_threshold", > "decision": "YES", > "explanation": "enough disk for shard on node, free: [22.4gb]" > }, > { > "decider": "throttling", > "decision": "YES", > "explanation": "below shard recovery limit of [2]" > }, > { > "decider": "replica\_after\_primary\_active", > "decision": "YES", > "explanation": "primary is already active" > }, > { > "decider": "snapshot\_in\_progress", > "decision": "YES", > "explanation": "shard not primary or relocation disabled" > } > ], > "parameters": { > "allow\_primary": false, > "index": "winlogbeat-2016.07.01", > "node": "ith-grs-sec-centos03", > "shard": 1 > } > } > ], > "state": { > "blocks": {}, > "master\_node": "KMqX1LEhR1ubRj5ixdiFLw", > "nodes": { > "KMqX1LEhR1ubRj5ixdiFLw": { > "attributes": { > "master": "true", > "rack": "ith-grs-sec-centos08" > }, > "name": "ith-grs-sec-centos08", > "transport\_address": "10.200.144.25:9300" > }, > "W4MbeuAOTC-UbLovyqi4QA": { > "attributes": { > "master": "true", > "rack": "ith-grs-sec-centos03" > }, > "name": "ith-grs-sec-centos03", > "transport\_address": "10.200.144.23:9300" > }, > "j017l3miS6iLYlD4fLyHng": { > "attributes": { > "master": "true", > "rack": "ith-grs-sec-centos04" > }, > "name": "ith-grs-sec-centos04", > "transport\_address": "10.200.144.27:9300" > } > }, > "routing\_nodes": { then there's info on all of my nodes and shards, i removed this information because repetitive and the output was very very big, the info on all unassigned replicas is: > ``` > { > "index": "winlogbeat-2016.07.01", > "node": null, > "primary": false, > "relocating_node": null, > "shard": 2, > "state": "UNASSIGNED", > "unassigned_info": { > "at": "2016-07-01T15:03:18.945Z", > "reason": "REPLICA_ADDED" > }, > "version": 14 > }, > > ``` after the command the shard was allocated. If you need more info or the complete output let me know ---
**Author:** ![ywelsch](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ywelsch/32/7751_2.png) [@ywelsch](https://discuss.elastic.co/u/ywelsch)\ **Post date:** [July 4, 2016, 1:50pm UTC](https://discuss.elastic.co/t/problem-with-shards-when-upgrading-to-2-3-3/54639/4 "2016-07-04T13:50:40Z")
Are all the shards now allocated or just this one? Are shards allocated if you issue an empty reroute command? ```auto curl -XPOST 'localhost:9200/_cluster/reroute ``` ---
**Author:** ![misomij](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/misomij/32/10700_2.png) [@misomij](https://discuss.elastic.co/u/misomij)\ **Post date:** [July 4, 2016, 2:04pm UTC](https://discuss.elastic.co/t/problem-with-shards-when-upgrading-to-2-3-3/54639/5 "2016-07-04T14:04:03Z")
> [@ywelsch](#): > > Are all the shards now allocated or just this one? Only one shard was allocated > [@ywelsch](#): > > Are shards allocated if you issue an empty reroute command? No, shards are not allocated, i launched the command with explain but got no explanations > { > "acknowledged": true, > "explanations": , > "state": { > "blocks": {}, > "master\_node": "KMqX1LEhR1ubRj5ixdiFLw", > "nodes": { > "KMqX1LEhR1ubRj5ixdiFLw": { > "attributes": { > "master": "true", > "rack": "ith-grs-sec-centos08" > }, > "name": "ith-grs-sec-centos08", > "transport\_address": "10.200.144.25:9300" > }, > "W4MbeuAOTC-UbLovyqi4QA": { > "attributes": { > "master": "true", > "rack": "ith-grs-sec-centos03" > }, > "name": "ith-grs-sec-centos03", > "transport\_address": "10.200.144.23:9300" > }, > "j017l3miS6iLYlD4fLyHng": { > "attributes": { > "master": "true", > "rack": "ith-grs-sec-centos04" > }, > "name": "ith-grs-sec-centos04", > "transport\_address": "10.200.144.27:9300" > } > }, > "routing\_nodes": { ---
**Author:** ![ywelsch](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ywelsch/32/7751_2.png) [@ywelsch](https://discuss.elastic.co/u/ywelsch)\ **Post date:** [July 4, 2016, 2:14pm UTC](https://discuss.elastic.co/t/problem-with-shards-when-upgrading-to-2-3-3/54639/6 "2016-07-04T14:14:24Z")
Maybe allocation is just disabled? Can you show me the cluster settings? ```auto curl -XGET 'http://localhost:9200/_cluster/settings?pretty' ``` ---
**Author:** ![misomij](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/misomij/32/10700_2.png) [@misomij](https://discuss.elastic.co/u/misomij)\ **Post date:** [July 4, 2016, 2:56pm UTC](https://discuss.elastic.co/t/problem-with-shards-when-upgrading-to-2-3-3/54639/7 "2016-07-04T14:56:05Z")
> [@ywelsch](#): > > Can you show me the cluster settings? > > curl -XGET '[http://localhost:9200/\_cluster/settings?pretty](http://localhost:9200/_cluster/settings?pretty)' I had tried ad got > { > "persistent" : { }, > "transient" : { } > } I tried to enable with: > curl -XPUT 'localhost:9200/\_cluster/settings' -d '{ > "transient" : { > "cluster.routing.allocation.enable" : "all" > } > }' and now i get: > { > "persistent" : { }, > "transient" : { > "cluster" : { > "routing" : { > "allocation" : { > "enable" : "all" > } > } > } > } > } but after waiting a while the shards are not allocated yet 😕 ---
**Author:** ![ywelsch](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ywelsch/32/7751_2.png) [@ywelsch](https://discuss.elastic.co/u/ywelsch)\ **Post date:** [July 4, 2016, 3:50pm UTC](https://discuss.elastic.co/t/problem-with-shards-when-upgrading-to-2-3-3/54639/8 "2016-07-04T15:50:11Z")
I'm running out of ideas here. Can you check that all three nodes are indeed on v2.3.3? Also output of ```auto curl localhost:9200/_cluster/health?pretty ``` and ```auto curl localhost:9200/_cat/shards ``` ---
**Author:** ![misomij](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/misomij/32/10700_2.png) [@misomij](https://discuss.elastic.co/u/misomij)\ **Post date:** [July 4, 2016, 4:09pm UTC](https://discuss.elastic.co/t/problem-with-shards-when-upgrading-to-2-3-3/54639/9 "2016-07-04T16:09:25Z")
> [@ywelsch](#): > > Can you check that all three nodes are indeed on v2.3.3? I checked with **\_nodes?pretty** and I got that all versions are 2.3.3 > "KMqX1LEhR1ubRj5ixdiFLw" : { > "name" : "ith-grs-sec-centos08", > "transport\_address" : "10.200.144.25:9300", > "host" : "10.200.144.25", > "ip" : "10.200.144.25", > "version" : "2.3.3", > "build" : "218bdf1", > "http\_address" : "10.200.144.25:9200", > "attributes" : { > "rack" : "ith-grs-sec-centos08", > "master" : "true" > }, > ... > "W4MbeuAOTC-UbLovyqi4QA" : { > "name" : "ith-grs-sec-centos03", > "transport\_address" : "10.200.144.23:9300", > "host" : "10.200.144.23", > "ip" : "10.200.144.23", > "version" : "2.3.3", > "build" : "218bdf1", > "http\_address" : "10.200.144.23:9200", > "attributes" : { > "rack" : "ith-grs-sec-centos03", > "master" : "true" > }, > ... > "j017l3miS6iLYlD4fLyHng" : { > "name" : "ith-grs-sec-centos04", > "transport\_address" : "10.200.144.27:9300", > "host" : "10.200.144.27", > "ip" : "10.200.144.27", > "version" : "2.3.3", > "build" : "218bdf1", > "http\_address" : "10.200.144.27:9200", > "attributes" : { > "rack" : "ith-grs-sec-centos04", > "master" : "true" > }, If you need entire output just let me know. > [@ywelsch](#): > > Also output of > > curl localhost:9200/\_cluster/health?pretty Here it is: > { > "cluster\_name" : "security-test-kibana", > "status" : "yellow", > "timed\_out" : false, > "number\_of\_nodes" : 3, > "number\_of\_data\_nodes" : 3, > "active\_primary\_shards" : 495, > "active\_shards" : 595, > "relocating\_shards" : 0, > "initializing\_shards" : 0, > "unassigned\_shards" : 395, > "delayed\_unassigned\_shards" : 0, > "number\_of\_pending\_tasks" : 0, > "number\_of\_in\_flight\_fetch" : 0, > "task\_max\_waiting\_in\_queue\_millis" : 0, > "active\_shards\_percent\_as\_number" : 60.1010101010101 > } and finally: > [@ywelsch](#): > > and > > curl localhost:9200/\_cat/shards the output is too big for this form, I'm copying here some sample output lines: > winlogbeat-2016.06.30 3 p STARTED 6531 4.4mb 10.200.144.25 ith-grs-sec-centos08 > winlogbeat-2016.06.30 3 r UNASSIGNED > winlogbeat-2016.06.30 2 p STARTED 6372 4.4mb 10.200.144.25 ith-grs-sec-centos08 > winlogbeat-2016.06.30 2 r UNASSIGNED > winlogbeat-2016.06.30 1 p STARTED 6397 4.3mb 10.200.144.23 ith-grs-sec-centos03 > winlogbeat-2016.06.30 1 r UNASSIGNED > winlogbeat-2016.06.30 0 p STARTED 6235 4.3mb 10.200.144.25 ith-grs-sec-centos08 > winlogbeat-2016.06.30 0 r UNASSIGNED > winlogbeat-2016.07.01 3 p STARTED 5802 4.1mb 10.200.144.23 ith-grs-sec-centos03 > winlogbeat-2016.07.01 3 r UNASSIGNED > winlogbeat-2016.07.01 2 p STARTED 4394 2.9mb 10.200.144.25 ith-grs-sec-centos08 > winlogbeat-2016.07.01 2 r STARTED 4394 2.9mb 10.200.144.27 ith-grs-sec-centos04 > winlogbeat-2016.07.01 1 r STARTED 4517 3.1mb 10.200.144.23 ith-grs-sec-centos03 > winlogbeat-2016.07.01 1 p STARTED 4517 3.1mb 10.200.144.25 ith-grs-sec-centos08 > winlogbeat-2016.07.01 0 r STARTED 4461 3mb 10.200.144.23 ith-grs-sec-centos03 > winlogbeat-2016.07.01 0 p STARTED 4461 3mb 10.200.144.25 ith-grs-sec-centos08 > winlogbeat-2016.07.02 3 r STARTED 3085 2mb 10.200.144.25 ith-grs-sec-centos08 > winlogbeat-2016.07.02 3 p STARTED 3085 2mb 10.200.144.27 ith-grs-sec-centos04 > winlogbeat-2016.07.02 2 r STARTED 3108 2.1mb 10.200.144.23 ith-grs-sec-centos03 > winlogbeat-2016.07.02 2 p STARTED 3108 2.1mb 10.200.144.27 ith-grs-sec-centos04 > winlogbeat-2016.07.02 1 r STARTED 3165 2.1mb 10.200.144.23 ith-grs-sec-centos03 > winlogbeat-2016.07.02 1 p STARTED 3165 2.1mb 10.200.144.27 ith-grs-sec-centos04 > winlogbeat-2016.07.02 0 r STARTED 3194 2.1mb 10.200.144.23 ith-grs-sec-centos03 > winlogbeat-2016.07.02 0 p STARTED 3194 2.1mb 10.200.144.27 ith-grs-sec-centos04 The shards from 07.02 and after are those allocated after the upgrade. The started shards from 07.01 are those which i allocated with the command @ywelsch suggested to me. The shards from 07.01 and before are all like these, with primary allocated and replicas unassigned. If you need complete output let me know. ---
**Author:** ![ywelsch](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ywelsch/32/7751_2.png) [@ywelsch](https://discuss.elastic.co/u/ywelsch)\ **Post date:** [July 4, 2016, 5:07pm UTC](https://discuss.elastic.co/t/problem-with-shards-when-upgrading-to-2-3-3/54639/10 "2016-07-04T17:07:11Z")
Maybe it's easier to provide me with the full output of `curl 'http://localhost:9200/_cluster/state?pretty'` (upload on [http://pastebin.com](http://pastebin.com) for example). If it contains sensitive information, send me the link via private message here. ---
**Author:** ![misomij](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/misomij/32/10700_2.png) [@misomij](https://discuss.elastic.co/u/misomij)\ **Post date:** [July 5, 2016, 7:44am UTC](https://discuss.elastic.co/t/problem-with-shards-when-upgrading-to-2-3-3/54639/11 "2016-07-05T07:44:16Z")
Ok i uploaded the result on my google drive because it was too big also for pastebin. [https://drive.google.com/open?id=0B5Vf-vwufarxdU5jQXI0YXJGT3c](https://drive.google.com/open?id=0B5Vf-vwufarxdU5jQXI0YXJGT3c) ---
**Author:** ![ywelsch](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ywelsch/32/7751_2.png) [@ywelsch](https://discuss.elastic.co/u/ywelsch)\ **Post date:** [July 5, 2016, 8:42am UTC](https://discuss.elastic.co/t/problem-with-shards-when-upgrading-to-2-3-3/54639/12 "2016-07-05T08:42:22Z")
Ok, this shows the issue clearly. For many of the indices you have the index setting `index.routing.allocation.disable_allocation` set to `all`, disabling allocation for shards of that index. Please set this to `false` and shards should be allocating again. ```auto curl -XPUT http://localhost:9200/*/_settings -d '{"index.routing.allocation.disable_allocation": false}' ``` ---
**Author:** ![misomij](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/misomij/32/10700_2.png) [@misomij](https://discuss.elastic.co/u/misomij)\ **Post date:** [July 5, 2016, 9:52am UTC](https://discuss.elastic.co/t/problem-with-shards-when-upgrading-to-2-3-3/54639/13 "2016-07-05T09:52:12Z")
OK now is slowly allocating all the shards, It has not finished yet but the problem seems resolved. I can't understand why the setting was like this since I tell the cluster to enable allocation: > [@misomij](#): > > I tried to enable with: > > curl -XPUT 'localhost:9200/\_cluster/settings' -d '{ > "transient" : { > "cluster.routing.allocation.enable" : "all" > } > }' Anyway thank you very much! ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 5, 2017, 10:38pm UTC](https://discuss.elastic.co/t/problem-with-shards-when-upgrading-to-2-3-3/54639/14 "2017-07-05T22:38:04Z")