No failover if number_of_replicas exceeds number of nodes?

Hi all,

I've encountered some unexpected behaviour during my DR testing which I'm
trying to explain.

I have a 3-node geographically-separated cluster with the following
settings:

  • index.number_of_shards=5
  • index.number_of_replicas=2
  • discovery.zen.minimum_master_nodes: 2

I use number_of_replicas=2 for durability, so that each node will contain a
full set to data (meaning I can lose 2 of my 3 nodes without losing any
data).

However I am finding that if I shut down 2 nodes, after adjusting
minimum_master_nodes on the remaining node to 1 and restarting that node,
the cluster stays yellow with all shards unassigned. They remain in the
unassigned state until I manually reduce number_of_replicas to down 1 or 0.
Once number_of_replicas <= number of nodes, the shards reassign and the
cluster goes green. Just wondering if this behaviour is as designed?

Regards,
Mat

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/abb0e7cd-c11a-4914-a98f-70d68c1d17be%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

It's not recommended to run an Elasticsearch cluster across geographically
dispersed locations.

You cannot assign both primaries and replicas to a single node, it defeats
the purpose! So it's as design.

On 7 January 2015 at 14:08, Mathew D mathew.degerholm@gmail.com wrote:

Hi all,

I've encountered some unexpected behaviour during my DR testing which I'm
trying to explain.

I have a 3-node geographically-separated cluster with the following
settings:

  • index.number_of_shards=5
  • index.number_of_replicas=2
  • discovery.zen.minimum_master_nodes: 2

I use number_of_replicas=2 for durability, so that each node will contain
a full set to data (meaning I can lose 2 of my 3 nodes without losing any
data).

However I am finding that if I shut down 2 nodes, after adjusting
minimum_master_nodes on the remaining node to 1 and restarting that node,
the cluster stays yellow with all shards unassigned. They remain in the
unassigned state until I manually reduce number_of_replicas to down 1 or
0. Once number_of_replicas <= number of nodes, the shards reassign and the
cluster goes green. Just wondering if this behaviour is as designed?

Regards,
Mat

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/abb0e7cd-c11a-4914-a98f-70d68c1d17be%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/abb0e7cd-c11a-4914-a98f-70d68c1d17be%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEYi1X99BqEA%2BbkAF63K9OygEgU84BD-YJV7rwQ0YkXjm-VFwg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

I understand there's no point assigning primaries and replicas to a
single node, but in my case ES won't even allocate a primary (until I
reduce the number of replicas)

On Wednesday, January 7, 2015 4:58:58 PM UTC+13, Mark Walkom wrote:

It's not recommended to run an Elasticsearch cluster across geographically
dispersed locations.

You cannot assign both primaries and replicas to a single node, it defeats
the purpose! So it's as design.

On 7 January 2015 at 14:08, Mathew D <mathew.d...@gmail.com <javascript:>>
wrote:

Hi all,

I've encountered some unexpected behaviour during my DR testing which I'm
trying to explain.

I have a 3-node geographically-separated cluster with the following
settings:

  • index.number_of_shards=5
  • index.number_of_replicas=2
  • discovery.zen.minimum_master_nodes: 2

I use number_of_replicas=2 for durability, so that each node will contain
a full set to data (meaning I can lose 2 of my 3 nodes without losing any
data).

However I am finding that if I shut down 2 nodes, after adjusting
minimum_master_nodes on the remaining node to 1 and restarting that node,
the cluster stays yellow with all shards unassigned. They remain in the
unassigned state until I manually reduce number_of_replicas to down 1 or
0. Once number_of_replicas <= number of nodes, the shards reassign and the
cluster goes green. Just wondering if this behaviour is as designed?

Regards,
Mat

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/abb0e7cd-c11a-4914-a98f-70d68c1d17be%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/abb0e7cd-c11a-4914-a98f-70d68c1d17be%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/74ae6254-e4c5-4504-8bee-cd337666bea4%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

A cluster cannot be yellow if any primaries are unassigned. Are you sure
it's yellow before you set replica's to 0?

On 8 January 2015 at 06:52, Mathew D mathew.degerholm@gmail.com wrote:

I understand there's no point assigning primaries and replicas to a
single node, but in my case ES won't even allocate a primary (until I
reduce the number of replicas)

On Wednesday, January 7, 2015 4:58:58 PM UTC+13, Mark Walkom wrote:

It's not recommended to run an Elasticsearch cluster across
geographically dispersed locations.

You cannot assign both primaries and replicas to a single node, it
defeats the purpose! So it's as design.

On 7 January 2015 at 14:08, Mathew D mathew.d...@gmail.com wrote:

Hi all,

I've encountered some unexpected behaviour during my DR testing which
I'm trying to explain.

I have a 3-node geographically-separated cluster with the following
settings:

  • index.number_of_shards=5
  • index.number_of_replicas=2
  • discovery.zen.minimum_master_nodes: 2

I use number_of_replicas=2 for durability, so that each node will
contain a full set to data (meaning I can lose 2 of my 3 nodes without
losing any data).

However I am finding that if I shut down 2 nodes, after adjusting
minimum_master_nodes on the remaining node to 1 and restarting that node,
the cluster stays yellow with all shards unassigned. They remain in the
unassigned state until I manually reduce number_of_replicas to down 1 or
0. Once number_of_replicas <= number of nodes, the shards reassign and the
cluster goes green. Just wondering if this behaviour is as designed?

Regards,
Mat

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/abb0e7cd-c11a-4914-a98f-70d68c1d17be%
40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/abb0e7cd-c11a-4914-a98f-70d68c1d17be%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/74ae6254-e4c5-4504-8bee-cd337666bea4%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/74ae6254-e4c5-4504-8bee-cd337666bea4%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEYi1X8HPUQada0aLLGufES6PA05aMLrgHtHvvMmARwE6zFK-A%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Good point... actually it looks like the 'head' plugin is misreporting a
yellow status when replicas=2:

https://lh4.googleusercontent.com/-nv6MfIzqKuk/VK2x3_VXgsI/AAAAAAAAASI/EzZ5Ejrw4yo/s1600/cluster-status-with-2-replicas.png
Because when I access the _cluster/health endpoint, a status of red is
returned:

{
"cluster_name": "KR_elasticsearch_PROD",
"status": "red",
"timed_out": false,
"number_of_nodes": 1,
"number_of_data_nodes": 1,
"active_primary_shards": 0,
"active_shards": 0,
"relocating_shards": 0,
"initializing_shards": 0,
"unassigned_shards": 75
}

So you're correct that the cluster cannot be yellow with primaries
unassigned. However would still be good to know why ES would refuse to
allocate primary shards if the number of replicas exceeds the number of
nodes.

Cheers,
Mat

On Thursday, January 8, 2015 10:03:51 AM UTC+13, Mark Walkom wrote:

A cluster cannot be yellow if any primaries are unassigned. Are you sure
it's yellow before you set replica's to 0?

On 8 January 2015 at 06:52, Mathew D <mathew.d...@gmail.com <javascript:>>
wrote:

I understand there's no point assigning primaries and replicas to a
single node, but in my case ES won't even allocate a primary (until I
reduce the number of replicas)

On Wednesday, January 7, 2015 4:58:58 PM UTC+13, Mark Walkom wrote:

It's not recommended to run an Elasticsearch cluster across
geographically dispersed locations.

You cannot assign both primaries and replicas to a single node, it
defeats the purpose! So it's as design.

On 7 January 2015 at 14:08, Mathew D mathew.d...@gmail.com wrote:

Hi all,

I've encountered some unexpected behaviour during my DR testing which
I'm trying to explain.

I have a 3-node geographically-separated cluster with the following
settings:

  • index.number_of_shards=5
  • index.number_of_replicas=2
  • discovery.zen.minimum_master_nodes: 2

I use number_of_replicas=2 for durability, so that each node will
contain a full set to data (meaning I can lose 2 of my 3 nodes without
losing any data).

However I am finding that if I shut down 2 nodes, after adjusting
minimum_master_nodes on the remaining node to 1 and restarting that node,
the cluster stays yellow with all shards unassigned. They remain in the
unassigned state until I manually reduce number_of_replicas to down 1 or
0. Once number_of_replicas <= number of nodes, the shards reassign and the
cluster goes green. Just wondering if this behaviour is as designed?

Regards,
Mat

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/abb0e7cd-c11a-4914-a98f-70d68c1d17be%
40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/abb0e7cd-c11a-4914-a98f-70d68c1d17be%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/74ae6254-e4c5-4504-8bee-cd337666bea4%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/74ae6254-e4c5-4504-8bee-cd337666bea4%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/fbbf2012-5d5f-42f4-b388-762a997b8fc9%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

It seems highly unusual that this is occurring. I'd recommend that you open
a github issue with details and see what the devs think.

On 8 January 2015 at 09:38, Mathew D mathew.degerholm@gmail.com wrote:

Good point... actually it looks like the 'head' plugin is misreporting a
yellow status when replicas=2:

https://lh4.googleusercontent.com/-nv6MfIzqKuk/VK2x3_VXgsI/AAAAAAAAASI/EzZ5Ejrw4yo/s1600/cluster-status-with-2-replicas.png
Because when I access the _cluster/health endpoint, a status of red is
returned:

{
"cluster_name": "KR_elasticsearch_PROD",
"status": "red",
"timed_out": false,
"number_of_nodes": 1,
"number_of_data_nodes": 1,
"active_primary_shards": 0,
"active_shards": 0,
"relocating_shards": 0,
"initializing_shards": 0,
"unassigned_shards": 75
}

So you're correct that the cluster cannot be yellow with primaries
unassigned. However would still be good to know why ES would refuse to
allocate primary shards if the number of replicas exceeds the number of
nodes.

Cheers,
Mat

On Thursday, January 8, 2015 10:03:51 AM UTC+13, Mark Walkom wrote:

A cluster cannot be yellow if any primaries are unassigned. Are you sure
it's yellow before you set replica's to 0?

On 8 January 2015 at 06:52, Mathew D mathew.d...@gmail.com wrote:

I understand there's no point assigning primaries and replicas to a
single node, but in my case ES won't even allocate a primary (until I
reduce the number of replicas)

On Wednesday, January 7, 2015 4:58:58 PM UTC+13, Mark Walkom wrote:

It's not recommended to run an Elasticsearch cluster across
geographically dispersed locations.

You cannot assign both primaries and replicas to a single node, it
defeats the purpose! So it's as design.

On 7 January 2015 at 14:08, Mathew D mathew.d...@gmail.com wrote:

Hi all,

I've encountered some unexpected behaviour during my DR testing which
I'm trying to explain.

I have a 3-node geographically-separated cluster with the following
settings:

  • index.number_of_shards=5
  • index.number_of_replicas=2
  • discovery.zen.minimum_master_nodes: 2

I use number_of_replicas=2 for durability, so that each node will
contain a full set to data (meaning I can lose 2 of my 3 nodes without
losing any data).

However I am finding that if I shut down 2 nodes, after adjusting
minimum_master_nodes on the remaining node to 1 and restarting that node,
the cluster stays yellow with all shards unassigned. They remain in the
unassigned state until I manually reduce number_of_replicas to down 1 or
0. Once number_of_replicas <= number of nodes, the shards reassign and the
cluster goes green. Just wondering if this behaviour is as designed?

Regards,
Mat

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/abb0e7cd-c11a-4914-a98f-70d68c1d17be%40goo
glegroups.com
https://groups.google.com/d/msgid/elasticsearch/abb0e7cd-c11a-4914-a98f-70d68c1d17be%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/74ae6254-e4c5-4504-8bee-cd337666bea4%
40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/74ae6254-e4c5-4504-8bee-cd337666bea4%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/fbbf2012-5d5f-42f4-b388-762a997b8fc9%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/fbbf2012-5d5f-42f4-b388-762a997b8fc9%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEYi1X-FPiG_zbhvykqFUL%3DSJv9v6_LCmT8fxBgDpUS9Gp9nWw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

I've logged this as
https://github.com/elasticsearch/elasticsearch/issues/9213. Will mark this
thread as complete in favour of the github issue.

On Friday, January 9, 2015 at 12:41:26 PM UTC+13, Mark Walkom wrote:

It seems highly unusual that this is occurring. I'd recommend that you
open a github issue with details and see what the devs think.

On 8 January 2015 at 09:38, Mathew D <mathew.d...@gmail.com <javascript:>>
wrote:

Good point... actually it looks like the 'head' plugin is misreporting a
yellow status when replicas=2:

https://lh4.googleusercontent.com/-nv6MfIzqKuk/VK2x3_VXgsI/AAAAAAAAASI/EzZ5Ejrw4yo/s1600/cluster-status-with-2-replicas.png
Because when I access the _cluster/health endpoint, a status of red is
returned:

{
"cluster_name": "KR_elasticsearch_PROD",
"status": "red",
"timed_out": false,
"number_of_nodes": 1,
"number_of_data_nodes": 1,
"active_primary_shards": 0,
"active_shards": 0,
"relocating_shards": 0,
"initializing_shards": 0,
"unassigned_shards": 75
}

So you're correct that the cluster cannot be yellow with primaries
unassigned. However would still be good to know why ES would refuse to
allocate primary shards if the number of replicas exceeds the number of
nodes.

Cheers,
Mat

On Thursday, January 8, 2015 10:03:51 AM UTC+13, Mark Walkom wrote:

A cluster cannot be yellow if any primaries are unassigned. Are you sure
it's yellow before you set replica's to 0?

On 8 January 2015 at 06:52, Mathew D mathew.d...@gmail.com wrote:

I understand there's no point assigning primaries and replicas to a
single node, but in my case ES won't even allocate a primary (until I
reduce the number of replicas)

On Wednesday, January 7, 2015 4:58:58 PM UTC+13, Mark Walkom wrote:

It's not recommended to run an Elasticsearch cluster across
geographically dispersed locations.

You cannot assign both primaries and replicas to a single node, it
defeats the purpose! So it's as design.

On 7 January 2015 at 14:08, Mathew D mathew.d...@gmail.com wrote:

Hi all,

I've encountered some unexpected behaviour during my DR testing which
I'm trying to explain.

I have a 3-node geographically-separated cluster with the following
settings:

  • index.number_of_shards=5
  • index.number_of_replicas=2
  • discovery.zen.minimum_master_nodes: 2

I use number_of_replicas=2 for durability, so that each node will
contain a full set to data (meaning I can lose 2 of my 3 nodes without
losing any data).

However I am finding that if I shut down 2 nodes, after adjusting
minimum_master_nodes on the remaining node to 1 and restarting that node,
the cluster stays yellow with all shards unassigned. They remain in the
unassigned state until I manually reduce number_of_replicas to down 1 or
0. Once number_of_replicas <= number of nodes, the shards reassign and the
cluster goes green. Just wondering if this behaviour is as designed?

Regards,
Mat

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/abb0e7cd-c11a-4914-a98f-70d68c1d17be%40goo
glegroups.com
https://groups.google.com/d/msgid/elasticsearch/abb0e7cd-c11a-4914-a98f-70d68c1d17be%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/74ae6254-e4c5-4504-8bee-cd337666bea4%
40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/74ae6254-e4c5-4504-8bee-cd337666bea4%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/fbbf2012-5d5f-42f4-b388-762a997b8fc9%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/fbbf2012-5d5f-42f4-b388-762a997b8fc9%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/91a20069-ab08-41a5-8c73-7674eb0a8128%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.