Been a real joy working with es, and today I'm trying to see if I can stump
es with a tricky config.
Say I have 2 data nodes that contain a collection of indices that were set
up with number_of_shards=1 and number_of_replicas=1.
Say I want to add a data node that only contains replicas of the existing
shards. My understanding is I can spin up the new data node and then update
the indices to auto_expand_replicas=0-all and the new data node will
acquire a replica of all the existing shards.
But, my struggle is to ensure that going forward when new indices are
created that this 3rd node does not get any primary shards.
The general goal is to have a small subset of data nodes that are allowed
to get primary (and replica) shards while another subset of data nodes are
only allowed to get replica shards.
I've been wrestling with auto_expand_replicas and the
[index,cluster].routing.allocation.enable settings and can't figure out how
to make it work.
Never mind. Anyone please correct me if I'm wrong, but after some thought I
think I've convinced myself that there is no need for such a setup. I think
it's ok for the 3rd node to get primary shards. Still have to test this,
but if I were to have auto_expand_replicas set up on all the indices and I
were to remove the node then the number of replicas would decrease by one
and a replica on the other 2 data nodes would be elected as a primary.
N
On Tuesday, July 8, 2014 12:16:04 PM UTC-4, Ned Campion wrote:
Hi All,
Been a real joy working with es, and today I'm trying to see if I can
stump es with a tricky config.
Say I have 2 data nodes that contain a collection of indices that were set
up with number_of_shards=1 and number_of_replicas=1.
Say I want to add a data node that only contains replicas of the existing
shards. My understanding is I can spin up the new data node and then update
the indices to auto_expand_replicas=0-all and the new data node will
acquire a replica of all the existing shards.
But, my struggle is to ensure that going forward when new indices are
created that this 3rd node does not get any primary shards.
The general goal is to have a small subset of data nodes that are allowed
to get primary (and replica) shards while another subset of data nodes are
only allowed to get replica shards.
I've been wrestling with auto_expand_replicas and the
[index,cluster].routing.allocation.enable settings and can't figure out how
to make it work.
There's not really a need as you've discovered.
ES is really good at managing distribution of shards, and unless you are
running specific hardware for storage (ie tiered storage across different
nodes) or you want rack awareness, then it's better to just let ES deal
with the allocation of primary/replicas.
Never mind. Anyone please correct me if I'm wrong, but after some thought
I think I've convinced myself that there is no need for such a setup. I
think it's ok for the 3rd node to get primary shards. Still have to test
this, but if I were to have auto_expand_replicas set up on all the indices
and I were to remove the node then the number of replicas would decrease by
one and a replica on the other 2 data nodes would be elected as a primary.
N
On Tuesday, July 8, 2014 12:16:04 PM UTC-4, Ned Campion wrote:
Hi All,
Been a real joy working with es, and today I'm trying to see if I can
stump es with a tricky config.
Say I have 2 data nodes that contain a collection of indices that were
set up with number_of_shards=1 and number_of_replicas=1.
Say I want to add a data node that only contains replicas of the existing
shards. My understanding is I can spin up the new data node and then update
the indices to auto_expand_replicas=0-all and the new data node will
acquire a replica of all the existing shards.
But, my struggle is to ensure that going forward when new indices are
created that this 3rd node does not get any primary shards.
The general goal is to have a small subset of data nodes that are allowed
to get primary (and replica) shards while another subset of data nodes are
only allowed to get replica shards.
I've been wrestling with auto_expand_replicas and the
[index,cluster].routing.allocation.enable settings and can't figure out
how to make it work.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.