Can a data node be set up to only be allocated replicas?


(Ned Campion) #1

Hi All,

Been a real joy working with es, and today I'm trying to see if I can stump
es with a tricky config.

Say I have 2 data nodes that contain a collection of indices that were set
up with number_of_shards=1 and number_of_replicas=1.

Say I want to add a data node that only contains replicas of the existing
shards. My understanding is I can spin up the new data node and then update
the indices to auto_expand_replicas=0-all and the new data node will
acquire a replica of all the existing shards.

But, my struggle is to ensure that going forward when new indices are
created that this 3rd node does not get any primary shards.

The general goal is to have a small subset of data nodes that are allowed
to get primary (and replica) shards while another subset of data nodes are
only allowed to get replica shards.

I've been wrestling with auto_expand_replicas and the
[index,cluster].routing.allocation.enable settings and can't figure out how
to make it work.

Any suggestions from the es masters out there?

Many thanks,
Ned

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/b4459e87-4df2-4667-97a8-10bf62360d5f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(Ned Campion) #2

Never mind. Anyone please correct me if I'm wrong, but after some thought I
think I've convinced myself that there is no need for such a setup. I think
it's ok for the 3rd node to get primary shards. Still have to test this,
but if I were to have auto_expand_replicas set up on all the indices and I
were to remove the node then the number of replicas would decrease by one
and a replica on the other 2 data nodes would be elected as a primary.

N

On Tuesday, July 8, 2014 12:16:04 PM UTC-4, Ned Campion wrote:

Hi All,

Been a real joy working with es, and today I'm trying to see if I can
stump es with a tricky config.

Say I have 2 data nodes that contain a collection of indices that were set
up with number_of_shards=1 and number_of_replicas=1.

Say I want to add a data node that only contains replicas of the existing
shards. My understanding is I can spin up the new data node and then update
the indices to auto_expand_replicas=0-all and the new data node will
acquire a replica of all the existing shards.

But, my struggle is to ensure that going forward when new indices are
created that this 3rd node does not get any primary shards.

The general goal is to have a small subset of data nodes that are allowed
to get primary (and replica) shards while another subset of data nodes are
only allowed to get replica shards.

I've been wrestling with auto_expand_replicas and the
[index,cluster].routing.allocation.enable settings and can't figure out how
to make it work.

Any suggestions from the es masters out there?

Many thanks,
Ned

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/2e8e7972-3cbc-4074-85a9-f31b6616f81b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(Mark Walkom) #3

There's not really a need as you've discovered.
ES is really good at managing distribution of shards, and unless you are
running specific hardware for storage (ie tiered storage across different
nodes) or you want rack awareness, then it's better to just let ES deal
with the allocation of primary/replicas.

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: markw@campaignmonitor.com
web: www.campaignmonitor.com

On 9 July 2014 03:12, Ned Campion nedcampion@gmail.com wrote:

Never mind. Anyone please correct me if I'm wrong, but after some thought
I think I've convinced myself that there is no need for such a setup. I
think it's ok for the 3rd node to get primary shards. Still have to test
this, but if I were to have auto_expand_replicas set up on all the indices
and I were to remove the node then the number of replicas would decrease by
one and a replica on the other 2 data nodes would be elected as a primary.

N

On Tuesday, July 8, 2014 12:16:04 PM UTC-4, Ned Campion wrote:

Hi All,

Been a real joy working with es, and today I'm trying to see if I can
stump es with a tricky config.

Say I have 2 data nodes that contain a collection of indices that were
set up with number_of_shards=1 and number_of_replicas=1.

Say I want to add a data node that only contains replicas of the existing
shards. My understanding is I can spin up the new data node and then update
the indices to auto_expand_replicas=0-all and the new data node will
acquire a replica of all the existing shards.

But, my struggle is to ensure that going forward when new indices are
created that this 3rd node does not get any primary shards.

The general goal is to have a small subset of data nodes that are allowed
to get primary (and replica) shards while another subset of data nodes are
only allowed to get replica shards.

I've been wrestling with auto_expand_replicas and the
[index,cluster].routing.allocation.enable settings and can't figure out
how to make it work.

Any suggestions from the es masters out there?

Many thanks,
Ned

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/2e8e7972-3cbc-4074-85a9-f31b6616f81b%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/2e8e7972-3cbc-4074-85a9-f31b6616f81b%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEM624aKQwKXzeMUrObfxkpL8HiS323R4%2BWFRtnk-VGWRqJ1MQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


(system) #4