Clarity for shard allocation disable/enable during upgrade

Elastic Stack Elasticsearch
1

I have 8 data nodes and 6 coordinator nodes in an active cluster running
1.2.1

I want to upgrade to 1.3.1

When
reading http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/setup-upgrade.html
the upgrade docs am I correct to assume:

a) disable shard allocation before doing anything

b) proceed to upgrade each node to 1.3.1

c) only after ALL nodes are @ 1.3.1 then I can re-enable shard allocation.

My question is that at some point during the upgrade of all the data nodes,
the shards on them will be "unassigned" and the cluster will not
function... correct?

So in other words running some nodes as 1.2.1 and others as 1.3.1 with
shard allocation enabled is NOT advised and in general cluster
un-availability is expected due to shards being in an unassigned state as
each data node is upgraded.

At least this is the behavior I see today, (not during an upgrade) when I
disable allocation and restart a node, those shards are unassigned until I
re-enable allocation

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/a7c8a15b-3e74-4c38-ab03-ff3e7d6fcd32%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

2

When you disable reallocation and then reboot a node, it should simply
re-enable the shards locally rather than trying to do so on other nodes.

However as you have noticed this doesn't actually occur and I have raised a
github issue to get this investigated, so feel free to comment on it as
well - Local shard initialisation after node restart with disabled allocation doesn't appear to work as expected · Issue #7117 · elastic/elasticsearch · GitHub

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: markw@campaignmonitor.com
web: www.campaignmonitor.com

On 11 August 2014 23:31, bitsofinfo.g@gmail.com wrote:

I have 8 data nodes and 6 coordinator nodes in an active cluster running
1.2.1

I want to upgrade to 1.3.1

When reading
Elasticsearch Platform — Find real-time answers at scale | Elastic
the upgrade docs am I correct to assume:

a) disable shard allocation before doing anything

b) proceed to upgrade each node to 1.3.1

c) only after ALL nodes are @ 1.3.1 then I can re-enable shard allocation.

My question is that at some point during the upgrade of all the data
nodes, the shards on them will be "unassigned" and the cluster will not
function... correct?

So in other words running some nodes as 1.2.1 and others as 1.3.1 with
shard allocation enabled is NOT advised and in general cluster
un-availability is expected due to shards being in an unassigned state as
each data node is upgraded.

At least this is the behavior I see today, (not during an upgrade) when I
disable allocation and restart a node, those shards are unassigned until I
re-enable allocation

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/a7c8a15b-3e74-4c38-ab03-ff3e7d6fcd32%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/a7c8a15b-3e74-4c38-ab03-ff3e7d6fcd32%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEM624Zz_WEenoivSnn8q0jgYe_MTiCs%2BOaueAxCtNrZR526jw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

3

On Monday, 11 August 2014 15:31:28 UTC+2, bitsof...@gmail.com wrote:

I have 8 data nodes and 6 coordinator nodes in an active cluster running
1.2.1

I want to upgrade to 1.3.1

When reading
Elasticsearch Platform — Find real-time answers at scale | Elastic
the upgrade docs am I correct to assume:

a) disable shard allocation before doing anything

b) proceed to upgrade each node to 1.3.1

c) only after ALL nodes are @ 1.3.1 then I can re-enable shard allocation.

My question is that at some point during the upgrade of all the data
nodes, the shards on them will be "unassigned" and the cluster will not
function... correct?

So in other words running some nodes as 1.2.1 and others as 1.3.1 with
shard allocation enabled is NOT advised and in general cluster
un-availability is expected due to shards being in an unassigned state as
each data node is upgraded.

At least this is the behavior I see today, (not during an upgrade) when I
disable allocation and restart a node, those shards are unassigned until I
re-enable allocation

No, the procedure outlined above is not correct and would indeed result in
unassigned shards, as you suspect. Instead, you should:

  1. Disable allocation
  2. Upgrade ONE node
  3. Reenable allocation
  4. Wait for green
  5. Repeat

Even when following the above process, you will likely end up with shards
being copied over from one node to another (once allocation has been
reenabled). After restart, a replica will only reuse the segments that are
exactly the same as those in the primary. However, because primaries and
replicas refresh, flush, and merge at different times, shards diverge from
each other over time. The longer it has been since a replica was copied
over from the primary, the fewer identical segments they will have in
common.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/11f7f772-9d86-4a84-a523-a047f4bb88b4%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

4

Mark - isn't the shard allocation all/none a cluster wide setting? Hence
why it does that on all nodes?

Clinton - What you said makes sense, however if that procedure is incorrect
then the official upgrade page on the elasticsearch site should be changed,
as it states

"When the process is complete on all nodes, you can re-enable shard
reallocation"

On Tuesday, August 12, 2014 4:04:28 AM UTC-4, Clinton Gormley wrote:

On Monday, 11 August 2014 15:31:28 UTC+2, bitsof...@gmail.com wrote:

I have 8 data nodes and 6 coordinator nodes in an active cluster running
1.2.1

I want to upgrade to 1.3.1

When reading
Elasticsearch Platform — Find real-time answers at scale | Elastic
the upgrade docs am I correct to assume:

a) disable shard allocation before doing anything

b) proceed to upgrade each node to 1.3.1

c) only after ALL nodes are @ 1.3.1 then I can re-enable shard allocation.

My question is that at some point during the upgrade of all the data
nodes, the shards on them will be "unassigned" and the cluster will not
function... correct?

So in other words running some nodes as 1.2.1 and others as 1.3.1 with
shard allocation enabled is NOT advised and in general cluster
un-availability is expected due to shards being in an unassigned state as
each data node is upgraded.

At least this is the behavior I see today, (not during an upgrade) when I
disable allocation and restart a node, those shards are unassigned until I
re-enable allocation

No, the procedure outlined above is not correct and would indeed result in
unassigned shards, as you suspect. Instead, you should:

  1. Disable allocation
  2. Upgrade ONE node
  3. Reenable allocation
  4. Wait for green
  5. Repeat

Even when following the above process, you will likely end up with shards
being copied over from one node to another (once allocation has been
reenabled). After restart, a replica will only reuse the segments that are
exactly the same as those in the primary. However, because primaries and
replicas refresh, flush, and merge at different times, shards diverge from
each other over time. The longer it has been since a replica was copied
over from the primary, the fewer identical segments they will have in
common.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/095cf279-5c51-49bc-8fb3-536e3160a9ec%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

5

Also, Clinton, per the upgrade page it states the below, so what you are
saying is that re-enabling allocation after each node is restarted (going
from 1.2.1 to 1.31) that the below will not apply (incompatibility)
because shards would be going from 1.2.1 to 1.3.1 vs the reverse Correct?

"Running multiple versions of Elasticsearch in the same cluster for any
length of time beyond that required for an upgrade is not supported, as
shard replication from the more recent version to the previous versions
will not work."

On Tuesday, August 12, 2014 4:04:28 AM UTC-4, Clinton Gormley wrote:

On Monday, 11 August 2014 15:31:28 UTC+2, bitsof...@gmail.com wrote:

I have 8 data nodes and 6 coordinator nodes in an active cluster running
1.2.1

I want to upgrade to 1.3.1

When reading
Elasticsearch Platform — Find real-time answers at scale | Elastic
the upgrade docs am I correct to assume:

a) disable shard allocation before doing anything

b) proceed to upgrade each node to 1.3.1

c) only after ALL nodes are @ 1.3.1 then I can re-enable shard allocation.

My question is that at some point during the upgrade of all the data
nodes, the shards on them will be "unassigned" and the cluster will not
function... correct?

So in other words running some nodes as 1.2.1 and others as 1.3.1 with
shard allocation enabled is NOT advised and in general cluster
un-availability is expected due to shards being in an unassigned state as
each data node is upgraded.

At least this is the behavior I see today, (not during an upgrade) when I
disable allocation and restart a node, those shards are unassigned until I
re-enable allocation

No, the procedure outlined above is not correct and would indeed result in
unassigned shards, as you suspect. Instead, you should:

  1. Disable allocation
  2. Upgrade ONE node
  3. Reenable allocation
  4. Wait for green
  5. Repeat

Even when following the above process, you will likely end up with shards
being copied over from one node to another (once allocation has been
reenabled). After restart, a replica will only reuse the segments that are
exactly the same as those in the primary. However, because primaries and
replicas refresh, flush, and merge at different times, shards diverge from
each other over time. The longer it has been since a replica was copied
over from the primary, the fewer identical segments they will have in
common.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/0dc5d597-3658-447f-94e0-b4b357fa4b08%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.