# Clarity for shard allocation disable/enable during upgrade **URL:** \ **Category:** Elasticsearch\ **Created:** [August 11, 2014, 1:31pm UTC](https://discuss.elastic.co/t/clarity-for-shard-allocation-disable-enable-during-upgrade/19195 "2014-08-11T13:31:28Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![bitsofinfo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bitsofinfo/32/459_2.png) [@bitsofinfo](https://discuss.elastic.co/u/bitsofinfo)\ **Post date:** [August 11, 2014, 1:31pm UTC](https://discuss.elastic.co/t/clarity-for-shard-allocation-disable-enable-during-upgrade/19195/1 "2014-08-11T13:31:28Z")
I have 8 data nodes and 6 coordinator nodes in an active cluster running 1.2.1 I want to upgrade to 1.3.1 When reading [http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/setup-upgrade.html](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/setup-upgrade.html) the upgrade docs am I correct to assume: a) disable shard allocation before doing anything b) proceed to upgrade each node to 1.3.1 c) only after ALL nodes are @ 1.3.1 then I can re-enable shard allocation. My question is that at some point during the upgrade of all the data nodes, the shards on them will be "unassigned" and the cluster will not function... correct? So in other words running some nodes as 1.2.1 and others as 1.3.1 with shard allocation _enabled_ is NOT advised and in general cluster un-availability is expected due to shards being in an unassigned state as each data node is upgraded. At least this is the behavior I see today, (not during an upgrade) when I disable allocation and restart a node, those shards are unassigned until I re-enable allocation -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/a7c8a15b-3e74-4c38-ab03-ff3e7d6fcd32%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/a7c8a15b-3e74-4c38-ab03-ff3e7d6fcd32%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [August 12, 2014, 12:49am UTC](https://discuss.elastic.co/t/clarity-for-shard-allocation-disable-enable-during-upgrade/19195/2 "2014-08-12T00:49:50Z")
When you disable reallocation and then reboot a node, it should simply re-enable the shards locally rather than trying to do so on other nodes. However as you have noticed this doesn't actually occur and I have raised a github issue to get this investigated, so feel free to comment on it as well - [Local shard initialisation after node restart with disabled allocation doesn't appear to work as expected · Issue #7117 · elastic/elasticsearch · GitHub](https://github.com/elasticsearch/elasticsearch/issues/7117) Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: [markw@campaignmonitor.com](mailto:markw@campaignmonitor.com) web: [www.campaignmonitor.com](http://www.campaignmonitor.com) On 11 August 2014 23:31, [bitsofinfo.g@gmail.com](mailto:bitsofinfo.g@gmail.com) wrote: > I have 8 data nodes and 6 coordinator nodes in an active cluster running > 1.2.1 > > I want to upgrade to 1.3.1 > > When reading > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/setup-upgrade.html) > the upgrade docs am I correct to assume: > > a) disable shard allocation before doing anything > > b) proceed to upgrade each node to 1.3.1 > > c) only after ALL nodes are @ 1.3.1 then I can re-enable shard allocation. > > My question is that at some point during the upgrade of all the data > nodes, the shards on them will be "unassigned" and the cluster will not > function... correct? > > So in other words running some nodes as 1.2.1 and others as 1.3.1 with > shard allocation _enabled_ is NOT advised and in general cluster > un-availability is expected due to shards being in an unassigned state as > each data node is upgraded. > > At least this is the behavior I see today, (not during an upgrade) when I > disable allocation and restart a node, those shards are unassigned until I > re-enable allocation > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/a7c8a15b-3e74-4c38-ab03-ff3e7d6fcd32%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/a7c8a15b-3e74-4c38-ab03-ff3e7d6fcd32%40googlegroups.com) > [https://groups.google.com/d/msgid/elasticsearch/a7c8a15b-3e74-4c38-ab03-ff3e7d6fcd32%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/a7c8a15b-3e74-4c38-ab03-ff3e7d6fcd32%40googlegroups.com?utm_medium=email&utm_source=footer) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAEM624Zz\_WEenoivSnn8q0jgYe\_MTiCs%2BOaueAxCtNrZR526jw%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEM624Zz_WEenoivSnn8q0jgYe_MTiCs%2BOaueAxCtNrZR526jw%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Clinton\_Gormley](https://avatars.discourse-cdn.com/v4/letter/c/50afbb/32.png) [@Clinton\_Gormley](https://discuss.elastic.co/u/Clinton_Gormley)\ **Post date:** [August 12, 2014, 8:04am UTC](https://discuss.elastic.co/t/clarity-for-shard-allocation-disable-enable-during-upgrade/19195/3 "2014-08-12T08:04:28Z")
On Monday, 11 August 2014 15:31:28 UTC+2, [bitsof...@gmail.com](mailto:bitsof...@gmail.com) wrote: > I have 8 data nodes and 6 coordinator nodes in an active cluster running > 1.2.1 > > I want to upgrade to 1.3.1 > > When reading > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/setup-upgrade.html) > the upgrade docs am I correct to assume: > > a) disable shard allocation before doing anything > > b) proceed to upgrade each node to 1.3.1 > > c) only after ALL nodes are @ 1.3.1 then I can re-enable shard allocation. > > My question is that at some point during the upgrade of all the data > nodes, the shards on them will be "unassigned" and the cluster will not > function... correct? > > So in other words running some nodes as 1.2.1 and others as 1.3.1 with > shard allocation _enabled_ is NOT advised and in general cluster > un-availability is expected due to shards being in an unassigned state as > each data node is upgraded. > > At least this is the behavior I see today, (not during an upgrade) when I > disable allocation and restart a node, those shards are unassigned until I > re-enable allocation No, the procedure outlined above is not correct and would indeed result in unassigned shards, as you suspect. Instead, you should: 1. Disable allocation 2. Upgrade ONE node 3. Reenable allocation 4. Wait for green 5. Repeat Even when following the above process, you will likely end up with shards being copied over from one node to another (once allocation has been reenabled). After restart, a replica will only reuse the segments that are exactly the same as those in the primary. However, because primaries and replicas refresh, flush, and merge at different times, shards diverge from each other over time. The longer it has been since a replica was copied over from the primary, the fewer identical segments they will have in common. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/11f7f772-9d86-4a84-a523-a047f4bb88b4%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/11f7f772-9d86-4a84-a523-a047f4bb88b4%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![bitsofinfo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bitsofinfo/32/459_2.png) [@bitsofinfo](https://discuss.elastic.co/u/bitsofinfo)\ **Post date:** [August 12, 2014, 2:18pm UTC](https://discuss.elastic.co/t/clarity-for-shard-allocation-disable-enable-during-upgrade/19195/4 "2014-08-12T14:18:15Z")
Mark - isn't the shard allocation all/none a cluster wide setting? Hence why it does that on all nodes? Clinton - What you said makes sense, however if that procedure is incorrect then the official upgrade page on the elasticsearch site should be changed, as it states "When the process is complete on all nodes, you can re-enable shard reallocation" On Tuesday, August 12, 2014 4:04:28 AM UTC-4, Clinton Gormley wrote: > On Monday, 11 August 2014 15:31:28 UTC+2, [bitsof...@gmail.com](mailto:bitsof...@gmail.com) wrote: > > > I have 8 data nodes and 6 coordinator nodes in an active cluster running > > 1.2.1 > > > > I want to upgrade to 1.3.1 > > > > When reading > > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/setup-upgrade.html) > > the upgrade docs am I correct to assume: > > > > a) disable shard allocation before doing anything > > > > b) proceed to upgrade each node to 1.3.1 > > > > c) only after ALL nodes are @ 1.3.1 then I can re-enable shard allocation. > > > > My question is that at some point during the upgrade of all the data > > nodes, the shards on them will be "unassigned" and the cluster will not > > function... correct? > > > > So in other words running some nodes as 1.2.1 and others as 1.3.1 with > > shard allocation _enabled_ is NOT advised and in general cluster > > un-availability is expected due to shards being in an unassigned state as > > each data node is upgraded. > > > > At least this is the behavior I see today, (not during an upgrade) when I > > disable allocation and restart a node, those shards are unassigned until I > > re-enable allocation > > No, the procedure outlined above is not correct and would indeed result in > unassigned shards, as you suspect. Instead, you should: > > 1. Disable allocation > 2. Upgrade ONE node > 3. Reenable allocation > 4. Wait for green > 5. Repeat > > Even when following the above process, you will likely end up with shards > being copied over from one node to another (once allocation has been > reenabled). After restart, a replica will only reuse the segments that are > exactly the same as those in the primary. However, because primaries and > replicas refresh, flush, and merge at different times, shards diverge from > each other over time. The longer it has been since a replica was copied > over from the primary, the fewer identical segments they will have in > common. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/095cf279-5c51-49bc-8fb3-536e3160a9ec%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/095cf279-5c51-49bc-8fb3-536e3160a9ec%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![bitsofinfo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bitsofinfo/32/459_2.png) [@bitsofinfo](https://discuss.elastic.co/u/bitsofinfo)\ **Post date:** [August 12, 2014, 2:22pm UTC](https://discuss.elastic.co/t/clarity-for-shard-allocation-disable-enable-during-upgrade/19195/5 "2014-08-12T14:22:14Z")
Also, Clinton, per the upgrade page it states the below, so what you are saying is that re-enabling allocation after each node is restarted (going from 1.2.1 to 1.31) that the below _will not_ apply (incompatibility) because shards would be going from 1.2.1 to 1.3.1 vs the reverse Correct? "Running multiple versions of Elasticsearch in the same cluster for any length of time beyond that required for an upgrade is not supported, as shard replication from the more recent version to the previous versions will not work." On Tuesday, August 12, 2014 4:04:28 AM UTC-4, Clinton Gormley wrote: > On Monday, 11 August 2014 15:31:28 UTC+2, [bitsof...@gmail.com](mailto:bitsof...@gmail.com) wrote: > > > I have 8 data nodes and 6 coordinator nodes in an active cluster running > > 1.2.1 > > > > I want to upgrade to 1.3.1 > > > > When reading > > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/setup-upgrade.html) > > the upgrade docs am I correct to assume: > > > > a) disable shard allocation before doing anything > > > > b) proceed to upgrade each node to 1.3.1 > > > > c) only after ALL nodes are @ 1.3.1 then I can re-enable shard allocation. > > > > My question is that at some point during the upgrade of all the data > > nodes, the shards on them will be "unassigned" and the cluster will not > > function... correct? > > > > So in other words running some nodes as 1.2.1 and others as 1.3.1 with > > shard allocation _enabled_ is NOT advised and in general cluster > > un-availability is expected due to shards being in an unassigned state as > > each data node is upgraded. > > > > At least this is the behavior I see today, (not during an upgrade) when I > > disable allocation and restart a node, those shards are unassigned until I > > re-enable allocation > > No, the procedure outlined above is not correct and would indeed result in > unassigned shards, as you suspect. Instead, you should: > > 1. Disable allocation > 2. Upgrade ONE node > 3. Reenable allocation > 4. Wait for green > 5. Repeat > > Even when following the above process, you will likely end up with shards > being copied over from one node to another (once allocation has been > reenabled). After restart, a replica will only reuse the segments that are > exactly the same as those in the primary. However, because primaries and > replicas refresh, flush, and merge at different times, shards diverge from > each other over time. The longer it has been since a replica was copied > over from the primary, the fewer identical segments they will have in > common. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/0dc5d597-3658-447f-94e0-b4b357fa4b08%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/0dc5d597-3658-447f-94e0-b4b357fa4b08%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 1:09am UTC](https://discuss.elastic.co/t/clarity-for-shard-allocation-disable-enable-during-upgrade/19195/6 "2017-07-06T01:09:10Z")