# Issues upgrading cluster from 0.19.8 to 0.19.10/11 **URL:** \ **Category:** Elasticsearch\ **Created:** [October 30, 2012, 2:27pm UTC](https://discuss.elastic.co/t/issues-upgrading-cluster-from-0-19-8-to-0-19-10-11/9501 "2012-10-30T14:27:10Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![James\_Dooley](https://avatars.discourse-cdn.com/v4/letter/j/a8b319/32.png) [@James\_Dooley](https://discuss.elastic.co/u/James_Dooley)\ **Post date:** [October 30, 2012, 2:27pm UTC](https://discuss.elastic.co/t/issues-upgrading-cluster-from-0-19-8-to-0-19-10-11/9501/1 "2012-10-30T14:27:10Z")
This weekend I attempted to update my cluster from 0.19.8 to 0.19.11 but ran in to issues at the very end when updating the master server. The issues I am seeing is that any new index created will not be assigned to any servers. Any degraded index will not attempt to recovery and will stay in that state even once the restarted server comes back up. I did attempt a full cluster restart and flush it did not seem to help. I do have some custom allocations in place, but I would not expect them to cause problems. I was able to temporarily resolve the issue by downgrading the master servers to 0.19.8 (later upgraded to 0.19.9 successfully) while keeping the storage nodes as 0.19.11. Just a bit of additional information I have nodes broken up with three different tags: frontend, storage and backup. By default the new indexes are assigned to the frontend nodes. \_template/default: { "default" : { "template" : "\*", "order" : 0, "settings" : { "index.compress" : "true", "index.routing.allocation.exclude.tag" : "backup,storage", "index.number\_of\_shards" : "4", "index.routing.allocation.total\_shards\_per\_node" : "4" }, "mappings" : { } } } \_cluster/settings: { "persistent" : { "indices.recovery.max\_size\_per\_sec" : "50mb", "cluster.routing.allocation.exclude.tag" : "" }, "transient" : { } } When I upgrade the master servers I do not see anything strange in the logs, everything comes up properly. As soon as I restart one of the data nodes the shards will remain unassigned, there are no messages in the cluster log file that shows anything going on other then nodes joining and leaving. When a master running 0.19.9 takes over the shards immediately start allocating properly. I looked at the change log for 0.19.10 but I do not see anything that jumps out being a probable cause, does anyone have any ideas on what may be happening? Also, I have attempted to force the index to use the frontend servers using index.routing.allocation.include.tag but it did not make a difference. -- ---
**Author:** ![James\_Dooley](https://avatars.discourse-cdn.com/v4/letter/j/a8b319/32.png) [@James\_Dooley](https://discuss.elastic.co/u/James_Dooley)\ **Post date:** [November 15, 2012, 5:48pm UTC](https://discuss.elastic.co/t/issues-upgrading-cluster-from-0-19-8-to-0-19-10-11/9501/2 "2012-11-15T17:48:47Z")
I did a little more testing today since I did not get any responses on this. DEBUG is showing messages like this: [2012-11-15 10:51:24,806][DEBUG][cluster.routing.allocation] [Frontend 1] [infuseds-2012.08.24][3] allocated on [[Storage 3][0gmkNk6HReiXV8udG7HBpQ][inet[/10.38.16.136:9300]]{tag=storage, master=false}], but can no longer be allocated on it, moving... [2012-11-15 10:51:24,807][DEBUG][cluster.routing.allocation] [Frontend 1] [infuseds-2012.08.24][3] can't move These messages only show up when the master node is running 0.19.10 or later. As soon as I downgrade to 0.19.9 everything allocates properly. I set up a test cluster and started adding the settings on the live cluster until I was able to reproduce the error. I was able to produce the error by setting cluster.routing.allocation.exclude.tag to "" in the persistent or transient cluster config. Once that is set I am unable to allocate any indexes on the entire cluster. As soon as I change the exclude line to something other then "" everything starts working again. Is there a way to completely remove the cluster.routing.allocation.exclude.tag value from \_cluster/settings? I see that it is being held in the nodes/0/\_state/global file but I would rather not mess with the file directly. -- ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [November 19, 2012, 12:51am UTC](https://discuss.elastic.co/t/issues-upgrading-cluster-from-0-19-8-to-0-19-10-11/9501/3 "2012-11-19T00:51:33Z")
Hi James, this is unfortunate, to me, it seems you were running into an undefined edge case where an exclude tag of "" can be interpreted as a "null" tag or a defined tag of length 0, a semantic which could have been changed recently. I wonder, does the cluster setting update API allow you to set cluster.routing.allocation.exclude.tag to null (the JSON null) so it vanishes? Best regards, Jörg -- ---
**Author:** ![James\_Carr](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/james_carr/32/956_2.png) [@James\_Carr](https://discuss.elastic.co/u/James_Carr)\ **Post date:** [November 19, 2012, 1:47am UTC](https://discuss.elastic.co/t/issues-upgrading-cluster-from-0-19-8-to-0-19-10-11/9501/4 "2012-11-19T01:47:26Z")
Is this addressed to me? Sent from my iPhone On Nov 18, 2012, at 6:51 PM, Jörg Prante [joergprante@gmail.com](mailto:joergprante@gmail.com) wrote: > Hi James, > > this is unfortunate, to me, it seems you were running into an undefined edge case where an exclude tag of "" can be interpreted as a "null" tag or a defined tag of length 0, a semantic which could have been changed recently. I wonder, does the cluster setting update API allow you to set cluster.routing.allocation.exclude.tag to null (the JSON null) so it vanishes? > > Best regards, > > Jörg > > -- -- ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 3:03am UTC](https://discuss.elastic.co/t/issues-upgrading-cluster-from-0-19-8-to-0-19-10-11/9501/5 "2017-07-06T03:03:46Z")
Closed on July 6, 2017, 3:03am UTC