Advanced Configuration

Elastic Stack Elasticsearch
1

We are currently running a diverse ES install, with 10+ servers
running in four different data centres. As such, I would like to
change the default sharding method to increase the number of replicas,
as I'd prefer that when a remote DC goes offline for any reason, I
don't end up with all the replicas for one shard at the end of that
now dead link.

Ideally, I'd like there to be one copy of each shard in each DC, so
four shards total, and weighted by server location.

I looked at the ES website but honestly, it's pretty thin on details
when it comes to configuring the server. Any tips would be
appreciated, thanks.

2

It's a bit difficult to find, but you can configure Elasticsearch to be
"rack aware" when doing shard allocation. See

http://www.elasticsearch.org/guide/reference/modules/cluster.html

for details. ES will even ensure that queries go to the same "rack" for
performance.

Cheers,
Dan

3

On top of that, it feels like forced allocation makes more sense in this
case.

On Fri, Apr 13, 2012 at 2:21 AM, Dan Everton dan@iocaine.org wrote:

It's a bit difficult to find, but you can configure Elasticsearch to be
"rack aware" when doing shard allocation. See

Elasticsearch Platform — Find real-time answers at scale | Elastic

for details. ES will even ensure that queries go to the same "rack" for
performance.

Cheers,
Dan

4

On Apr 12, 7:21 pm, Dan Everton d...@iocaine.org wrote:

Elasticsearch Platform — Find real-time answers at scale | Elastic

Okay, from that I got the following config:

for my UK DC:

node.zone: uk
cluster.routing.allocation.awareness.attributes: zone

For my local DC:

node.zone: local
cluster.routing.allocation.awareness.attributes: zone

So now there's a copy of each shard in each zone, but I'd like 2
copies of each shard in each zone. The docs suggest that the
following option should do this:

cluster.routing.allocation.cluster_concurrent_rebalance: 3

But adding it via curl hasn't changed the distribution of shards.

5

If you want 2 copies on each zone, then set the index.number_of_replicas to
3 (in addition to the primary, it means you will have 4 copies). I highly
suggest using forced allocation awareness in this case.

On Fri, Apr 13, 2012 at 6:38 PM, newsgroup@syslog.com
newsgroup@syslog.comwrote:

On Apr 12, 7:21 pm, Dan Everton d...@iocaine.org wrote:

Elasticsearch Platform — Find real-time answers at scale | Elastic

Okay, from that I got the following config:

for my UK DC:

node.zone: uk
cluster.routing.allocation.awareness.attributes: zone

For my local DC:

node.zone: local
cluster.routing.allocation.awareness.attributes: zone

So now there's a copy of each shard in each zone, but I'd like 2
copies of each shard in each zone. The docs suggest that the
following option should do this:

cluster.routing.allocation.cluster_concurrent_rebalance: 3

But adding it via curl hasn't changed the distribution of shards.