Cluster allocation awareness - opposite

Hi All,
we currently have an Elasticsearch (1.1.1) cluster distributed among DCs

  • 3 Data nodes in 3 DC
  • 1 Gateway node

each DC index its own data and no shards replica is happening between DC.
The gateway lets us to query all the indexes in all the DC.

Now, for performance and redundancy we would like to add one data node per
DC and replicate the 'local' indexes.

We read about cluster allocation awareness, and thought that was the
perfect solution, until we realize it actually acts opposite of what we
want.
If node members are in the same awareness attribute, they would just
'ignore' each others and don't spread shards or replicate.

Is such a deployment possible/does it makes sense?

Thanks a million,
Matteo

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/3cff8e6e-9b3b-432c-baba-03dd44b3862c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

It's a little unclear what you are doing.

You indicate you have a single cluster but then no replication happens
between the nodes in each DC?
By gateway do you mean a tribe node?
And by replicate the local indexes do you want to set replicas = 1 and then
have them on this new node in each DC?

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: markw@campaignmonitor.com
web: www.campaignmonitor.com

On 12 September 2014 22:40, spezam spezam@gmail.com wrote:

Hi All,
we currently have an Elasticsearch (1.1.1) cluster distributed among DCs

  • 3 Data nodes in 3 DC
  • 1 Gateway node

each DC index its own data and no shards replica is happening between DC.
The gateway lets us to query all the indexes in all the DC.

Now, for performance and redundancy we would like to add one data node per
DC and replicate the 'local' indexes.

We read about cluster allocation awareness, and thought that was the
perfect solution, until we realize it actually acts opposite of what we
want.
If node members are in the same awareness attribute, they would just
'ignore' each others and don't spread shards or replicate.

Is such a deployment possible/does it makes sense?

Thanks a million,
Matteo

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/3cff8e6e-9b3b-432c-baba-03dd44b3862c%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/3cff8e6e-9b3b-432c-baba-03dd44b3862c%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEM624Y-3rjNgomPDz24mCKRb0c5Fo6Hd8KKwRaoo8M%2BL%2BWPsQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Hi Mark,
precise, there's no replication happening between the nodes in the DC, they
are just aware of each others.

The gateway is a no-data node that is used by Kibana to query all the other
nodes in the cluster and return aggregated queries.

Yes, I would like to add a new node (node2) in DC1 (for example) and
replicate data between node1 and node2 in the same DC.

On Saturday, September 13, 2014 12:40:51 AM UTC+2, Mark Walkom wrote:

It's a little unclear what you are doing.

You indicate you have a single cluster but then no replication happens
between the nodes in each DC?
By gateway do you mean a tribe node?
And by replicate the local indexes do you want to set replicas = 1 and
then have them on this new node in each DC?

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: ma...@campaignmonitor.com <javascript:>
web: www.campaignmonitor.com

On 12 September 2014 22:40, spezam <spe...@gmail.com <javascript:>> wrote:

Hi All,
we currently have an Elasticsearch (1.1.1) cluster distributed among DCs

  • 3 Data nodes in 3 DC
  • 1 Gateway node

each DC index its own data and no shards replica is happening between DC.
The gateway lets us to query all the indexes in all the DC.

Now, for performance and redundancy we would like to add one data node
per DC and replicate the 'local' indexes.

We read about cluster allocation awareness, and thought that was the
perfect solution, until we realize it actually acts opposite of what we
want.
If node members are in the same awareness attribute, they would just
'ignore' each others and don't spread shards or replicate.

Is such a deployment possible/does it makes sense?

Thanks a million,
Matteo

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/3cff8e6e-9b3b-432c-baba-03dd44b3862c%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/3cff8e6e-9b3b-432c-baba-03dd44b3862c%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/4100dab3-e194-4834-94cb-6601d98194d0%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

At the end, I decided to divide the single cluster in clusterS and use the
new tribe capabilities.

it works really good.

On Monday, September 15, 2014 10:03:41 AM UTC+2, spezam wrote:

Hi Mark,
precise, there's no replication happening between the nodes in the DC,
they are just aware of each others.

The gateway is a no-data node that is used by Kibana to query all the
other nodes in the cluster and return aggregated queries.

Yes, I would like to add a new node (node2) in DC1 (for example) and
replicate data between node1 and node2 in the same DC.

On Saturday, September 13, 2014 12:40:51 AM UTC+2, Mark Walkom wrote:

It's a little unclear what you are doing.

You indicate you have a single cluster but then no replication happens
between the nodes in each DC?
By gateway do you mean a tribe node?
And by replicate the local indexes do you want to set replicas = 1 and
then have them on this new node in each DC?

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: ma...@campaignmonitor.com
web: www.campaignmonitor.com

On 12 September 2014 22:40, spezam spe...@gmail.com wrote:

Hi All,
we currently have an Elasticsearch (1.1.1) cluster distributed among DCs

  • 3 Data nodes in 3 DC
  • 1 Gateway node

each DC index its own data and no shards replica is happening between DC.
The gateway lets us to query all the indexes in all the DC.

Now, for performance and redundancy we would like to add one data node
per DC and replicate the 'local' indexes.

We read about cluster allocation awareness, and thought that was the
perfect solution, until we realize it actually acts opposite of what we
want.
If node members are in the same awareness attribute, they would just
'ignore' each others and don't spread shards or replicate.

Is such a deployment possible/does it makes sense?

Thanks a million,
Matteo

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/3cff8e6e-9b3b-432c-baba-03dd44b3862c%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/3cff8e6e-9b3b-432c-baba-03dd44b3862c%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/58116aca-41b8-4f43-9ad9-09f0ef67b072%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.