Configuring replica nodes at a global level


(Phani) #1

Again some newbie questions. I'm wondering how to configure a replica node
at a global level rather than at index level. These are the questions that
I have :

  1. Can we configure that nodes 2 & 3 are replicas of node 1 in ES ? If
    yes, how ? - This is because I don't want to explicitly configure the
    settings for index level and what ever the replicas are for a primary shard
    should automatically replicate the indexes/data that are part of the
    primary shard.
    1. I'm wondering if there is any property like node.replica = true
      and using the same node name in case of replicas will be the solution to
      this.
  2. We have index level replica settings for shards - Can we control on
    which nodes the index replicas exists (this is in some sense related to 1st
    point) ?

Thanks,
Phani.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(Boaz Leskes) #2

Hi Pahni,

I'm not 100% sure of what your are trying to achieve - but here are some
tools that would probably help. The thing to realise is that replicas are
copies of index shards and automatically spread out through the cluster to
balance usage. There is no notion of a replica node but rather a replica
shard.

That said, things you can do to control replica counts:

  1. Use index templates to control the default number of replicas
    (recommended)
    : http://www.elasticsearch.org/guide/reference/api/admin-indices-templates/
  2. Use elasticsearch.yml to control the default number of replicas (you
    have to make sure all nodes have the same yml)
  3. Use allocation filtering which nodes indices are allocated
    on: http://www.elasticsearch.org/guide/reference/index-modules/allocation/

Hope this helps,
Boaz

On Tuesday, September 24, 2013 7:47:51 PM UTC+2, Phani Chaitanya wrote:

Again some newbie questions. I'm wondering how to configure a replica node
at a global level rather than at index level. These are the questions that
I have :

  1. Can we configure that nodes 2 & 3 are replicas of node 1 in ES ? If
    yes, how ? - This is because I don't want to explicitly configure the
    settings for index level and what ever the replicas are for a primary shard
    should automatically replicate the indexes/data that are part of the
    primary shard.
    1. I'm wondering if there is any property like node.replica = true
      and using the same node name in case of replicas will be the solution to
      this.
  2. We have index level replica settings for shards - Can we control on
    which nodes the index replicas exists (this is in some sense related to 1st
    point) ?

Thanks,
Phani.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(Phani) #3

Hi Boaz,

Thanks for the pointers. I'll try to put my question in a different way.
May be I'm trying to draw analogies w.r.t SolrCloud which is why I'm
getting these questions. So, I'll start with the following question.

In SolrCloud when you say that you have 2 shards (each shard is a solr
server) and start 4 servers then two among those will be the replicas of
the 2 primary(leader) shards. May be it's wrong to compare with SolrCloud
but just wondering the shards to ES node mapping. An ES node can have more
than 1 shard ? (some of them can be primary shards & some can be replicas
based on the balancing done by ES ?).

And, I've a question on how the replicas of shards are decided. Lets say
I've configured that number of shards as 4 and number of replicas as 1
which I assume means that each shard will have 1 replica. Again in
SolrCloud, after starting that many servers that you specify as the number
of shards & the servers that you start later will automatically become
replicas. How are replica shards decided in ES ?

Also, for your 2nd point where you mentioned to make sure all nodes have
the same yml. Is it for the index level settings in yml ? because we can
change other parameters in yml which are node specific right ? like
node.data & node.master as true/false and the like.

Thanks,
Phani.

On Wed, Sep 25, 2013 at 3:18 AM, Boaz Leskes b.leskes@gmail.com wrote:

Hi Pahni,

I'm not 100% sure of what your are trying to achieve - but here are some
tools that would probably help. The thing to realise is that replicas are
copies of index shards and automatically spread out through the cluster to
balance usage. There is no notion of a replica node but rather a replica
shard.

That said, things you can do to control replica counts:

  1. Use index templates to control the default number of replicas
    (recommended) :
    http://www.elasticsearch.org/guide/reference/api/admin-indices-templates/
  2. Use elasticsearch.yml to control the default number of replicas (you
    have to make sure all nodes have the same yml)
  3. Use allocation filtering which nodes indices are allocated on:
    http://www.elasticsearch.org/guide/reference/index-modules/allocation/

Hope this helps,
Boaz

On Tuesday, September 24, 2013 7:47:51 PM UTC+2, Phani Chaitanya wrote:

Again some newbie questions. I'm wondering how to configure a replica
node at a global level rather than at index level. These are the questions
that I have :

  1. Can we configure that nodes 2 & 3 are replicas of node 1 in ES ?
    If yes, how ? - This is because I don't want to explicitly configure the
    settings for index level and what ever the replicas are for a primary shard
    should automatically replicate the indexes/data that are part of the
    primary shard.
    1. I'm wondering if there is any property like node.replica = true
      and using the same node name in case of replicas will be the solution to
      this.
  2. We have index level replica settings for shards - Can we control
    on which nodes the index replicas exists (this is in some sense related to
    1st point) ?

Thanks,
Phani.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/RgtLqTB3vvc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(Boaz Leskes) #4

Hi Phani,

I'm not too familiar with SolrCloud, so it's hard for me to give a good
answer. I suggest you watch this talk
http://www.elasticsearch.org/videos/big-data-search-and-analytics/ which
explains the distribution model with some examples.

As to your ES question - yes you can have multiple shards on a node. And
yes - the file doesn't need to be exactly the same - I meant indeed the
index settings.

Cheers,
Boaz

On Wed, Sep 25, 2013 at 7:01 PM, Phani Chaitanya Vempaty <
pvempaty.usc@gmail.com> wrote:

Hi Boaz,

Thanks for the pointers. I'll try to put my question in a different way.
May be I'm trying to draw analogies w.r.t SolrCloud which is why I'm
getting these questions. So, I'll start with the following question.

In SolrCloud when you say that you have 2 shards (each shard is a solr
server) and start 4 servers then two among those will be the replicas of
the 2 primary(leader) shards. May be it's wrong to compare with SolrCloud
but just wondering the shards to ES node mapping. An ES node can have more
than 1 shard ? (some of them can be primary shards & some can be replicas
based on the balancing done by ES ?).

And, I've a question on how the replicas of shards are decided. Lets say
I've configured that number of shards as 4 and number of replicas as 1
which I assume means that each shard will have 1 replica. Again in
SolrCloud, after starting that many servers that you specify as the number
of shards & the servers that you start later will automatically become
replicas. How are replica shards decided in ES ?

Also, for your 2nd point where you mentioned to make sure all nodes have
the same yml. Is it for the index level settings in yml ? because we can
change other parameters in yml which are node specific right ? like
node.data & node.master as true/false and the like.

Thanks,
Phani.

On Wed, Sep 25, 2013 at 3:18 AM, Boaz Leskes b.leskes@gmail.com wrote:

Hi Pahni,

I'm not 100% sure of what your are trying to achieve - but here are some
tools that would probably help. The thing to realise is that replicas are
copies of index shards and automatically spread out through the cluster to
balance usage. There is no notion of a replica node but rather a replica
shard.

That said, things you can do to control replica counts:

  1. Use index templates to control the default number of replicas
    (recommended) :
    http://www.elasticsearch.org/guide/reference/api/admin-indices-templates/
  2. Use elasticsearch.yml to control the default number of replicas (you
    have to make sure all nodes have the same yml)
  3. Use allocation filtering which nodes indices are allocated on:
    http://www.elasticsearch.org/guide/reference/index-modules/allocation/

Hope this helps,
Boaz

On Tuesday, September 24, 2013 7:47:51 PM UTC+2, Phani Chaitanya wrote:

Again some newbie questions. I'm wondering how to configure a replica
node at a global level rather than at index level. These are the questions
that I have :

  1. Can we configure that nodes 2 & 3 are replicas of node 1 in ES ?
    If yes, how ? - This is because I don't want to explicitly configure the
    settings for index level and what ever the replicas are for a primary shard
    should automatically replicate the indexes/data that are part of the
    primary shard.
    1. I'm wondering if there is any property like node.replica =
      true and using the same node name in case of replicas will be the solution
      to this.
  2. We have index level replica settings for shards - Can we control
    on which nodes the index replicas exists (this is in some sense related to
    1st point) ?

Thanks,
Phani.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/RgtLqTB3vvc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/RgtLqTB3vvc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(Phani) #5

Thanks Boaz for the answers and the link to useful information.

On Thu, Sep 26, 2013 at 12:21 AM, Boaz Leskes b.leskes@gmail.com wrote:

Hi Phani,

I'm not too familiar with SolrCloud, so it's hard for me to give a good
answer. I suggest you watch this talk
http://www.elasticsearch.org/videos/big-data-search-and-analytics/ which
explains the distribution model with some examples.

As to your ES question - yes you can have multiple shards on a node. And
yes - the file doesn't need to be exactly the same - I meant indeed the
index settings.

Cheers,
Boaz

On Wed, Sep 25, 2013 at 7:01 PM, Phani Chaitanya Vempaty <
pvempaty.usc@gmail.com> wrote:

Hi Boaz,

Thanks for the pointers. I'll try to put my question in a different
way. May be I'm trying to draw analogies w.r.t SolrCloud which is why I'm
getting these questions. So, I'll start with the following question.

In SolrCloud when you say that you have 2 shards (each shard is a solr
server) and start 4 servers then two among those will be the replicas of
the 2 primary(leader) shards. May be it's wrong to compare with SolrCloud
but just wondering the shards to ES node mapping. An ES node can have more
than 1 shard ? (some of them can be primary shards & some can be replicas
based on the balancing done by ES ?).

And, I've a question on how the replicas of shards are decided. Lets say
I've configured that number of shards as 4 and number of replicas as 1
which I assume means that each shard will have 1 replica. Again in
SolrCloud, after starting that many servers that you specify as the number
of shards & the servers that you start later will automatically become
replicas. How are replica shards decided in ES ?

Also, for your 2nd point where you mentioned to make sure all nodes have
the same yml. Is it for the index level settings in yml ? because we can
change other parameters in yml which are node specific right ? like
node.data & node.master as true/false and the like.

Thanks,
Phani.

On Wed, Sep 25, 2013 at 3:18 AM, Boaz Leskes b.leskes@gmail.com wrote:

Hi Pahni,

I'm not 100% sure of what your are trying to achieve - but here are some
tools that would probably help. The thing to realise is that replicas are
copies of index shards and automatically spread out through the cluster to
balance usage. There is no notion of a replica node but rather a replica
shard.

That said, things you can do to control replica counts:

  1. Use index templates to control the default number of replicas
    (recommended) :
    http://www.elasticsearch.org/guide/reference/api/admin-indices-templates/
  2. Use elasticsearch.yml to control the default number of replicas (you
    have to make sure all nodes have the same yml)
  3. Use allocation filtering which nodes indices are allocated on:
    http://www.elasticsearch.org/guide/reference/index-modules/allocation/

Hope this helps,
Boaz

On Tuesday, September 24, 2013 7:47:51 PM UTC+2, Phani Chaitanya wrote:

Again some newbie questions. I'm wondering how to configure a replica
node at a global level rather than at index level. These are the questions
that I have :

  1. Can we configure that nodes 2 & 3 are replicas of node 1 in ES ?
    If yes, how ? - This is because I don't want to explicitly configure the
    settings for index level and what ever the replicas are for a primary shard
    should automatically replicate the indexes/data that are part of the
    primary shard.
    1. I'm wondering if there is any property like node.replica =
      true and using the same node name in case of replicas will be the solution
      to this.
  2. We have index level replica settings for shards - Can we control
    on which nodes the index replicas exists (this is in some sense related to
    1st point) ?

Thanks,
Phani.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/RgtLqTB3vvc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/RgtLqTB3vvc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/RgtLqTB3vvc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(system) #6