We've recently set up a 5 node ES cluster, serving our 6-shards / 1-replica
index (we chose 6 back when we only had 3 nodes). We sometimes find a
highly uneven distribution of shards across the nodes. For example, when we
had 3 nodes, 4/6 of the index lived on 1 node, 2/6 lived on another, and
0/0 lived on the last node. While we do have distributed replicas, we've
noticed a very very high "OS load" on the machine that served 4/6 of the
index. In fact, the OS load was so high, that it brought the VM to
a screeching halt.
My question is: how can we force a more even distribution of the shards?
I've played around with "cluster.routing.allocation.balance.*", but those
had little to no effect. And, from my understanding of the docs,
"cluster.routing.allocation.awareness" is aimed for a more even zone
distribution, rather than shards in the same zone.
Thanks in advance for all your help! Much appreciated!
When we update these dynamically using the API we found you sometimes have
to jiggle the shard allocator to force it to start thinking again. So use
the manual allocation api to move a small shard from one node to another or
something.
We've recently set up a 5 node ES cluster, serving our 6-shards /
1-replica index (we chose 6 back when we only had 3 nodes). We sometimes
find a highly uneven distribution of shards across the nodes. For example,
when we had 3 nodes, 4/6 of the index lived on 1 node, 2/6 lived on
another, and 0/0 lived on the last node. While we do have distributed
replicas, we've noticed a very very high "OS load" on the machine that
served 4/6 of the index. In fact, the OS load was so high, that it brought
the VM to a screeching halt.
My question is: how can we force a more even distribution of the shards?
I've played around with "cluster.routing.allocation.balance.*", but those
had little to no effect. And, from my understanding of the docs,
"cluster.routing.allocation.awareness" is aimed for a more even zone
distribution, rather than shards in the same zone.
Thanks in advance for all your help! Much appreciated!
Thanks for that, Nik. I'm okay with evenly spreading all the indices,
rather than just the one I'm having issues with. I'll give your config a
try!
Def no special configurations on that one. We didn't even realize there was
such a thing as allocation configuration up until yesterday (after the 0/6
allocation happened). The 0/6 node did, however, contain 4/6 replicas... If
that makes a difference?
On Wednesday, July 23, 2014 9:03:46 AM UTC-4, Nikolas Everett wrote:
For the 0/6 node are you sure you don't have some configuration preventing
shards from allocating there?
We use this:
elasticsearch.yml.erb · operations-puppet
but its is designed to more evenly spread shards from every index across
all of our nodes rather than more evenly spread total shards across the
nodes.
When we update these dynamically using the API we found you sometimes have
to jiggle the shard allocator to force it to start thinking again. So use
the manual allocation api to move a small shard from one node to another or
something.
We've recently set up a 5 node ES cluster, serving our 6-shards /
1-replica index (we chose 6 back when we only had 3 nodes). We sometimes
find a highly uneven distribution of shards across the nodes. For example,
when we had 3 nodes, 4/6 of the index lived on 1 node, 2/6 lived on
another, and 0/0 lived on the last node. While we do have distributed
replicas, we've noticed a very very high "OS load" on the machine that
served 4/6 of the index. In fact, the OS load was so high, that it brought
the VM to a screeching halt.
My question is: how can we force a more even distribution of the shards?
I've played around with "cluster.routing.allocation.balance.*", but those
had little to no effect. And, from my understanding of the docs,
"cluster.routing.allocation.awareness" is aimed for a more even zone
distribution, rather than shards in the same zone.
Thanks in advance for all your help! Much appreciated!
Thanks for that, Nik. I'm okay with evenly spreading all the indices,
rather than just the one I'm having issues with. I'll give your config a
try!
Def no special configurations on that one. We didn't even realize there
was such a thing as allocation configuration up until yesterday (after the
0/6 allocation happened). The 0/6 node did, however, contain 4/6
replicas... If that makes a difference?
It certainly does. The way the allocation settings are configured now the
cluster thinks of replicas as just about the same as masters. For the most
part they are, too. For us they certainly are.
If you are sure that being a shard master is significantly more work then
being a shard replica then raise the "primary" setting to something like .5
and lower everything else to total 1.
That whole totally 1 thing isn't required but it makes the numbers easier
to think about.
Got it. To be honest, I was pretty sure of that, up until this AM, when
that same OS Load spike happened again. But this time, the shards were
allocated more evenly. So I'm not sure that's even the problem any more. I
just posted a new post with more information about the load spike issue.
Would you mind taking a look?
Thanks for all your help, Nik.
On Wednesday, July 23, 2014 9:30:00 AM UTC-4, Nikolas Everett wrote:
Thanks for that, Nik. I'm okay with evenly spreading all the indices,
rather than just the one I'm having issues with. I'll give your config a
try!
Def no special configurations on that one. We didn't even realize there
was such a thing as allocation configuration up until yesterday (after the
0/6 allocation happened). The 0/6 node did, however, contain 4/6
replicas... If that makes a difference?
It certainly does. The way the allocation settings are configured now the
cluster thinks of replicas as just about the same as masters. For the most
part they are, too. For us they certainly are.
If you are sure that being a shard master is significantly more work then
being a shard replica then raise the "primary" setting to something like .5
and lower everything else to total 1.
That whole totally 1 thing isn't required but it makes the numbers easier
to think about.
And it doesn't seem to do much... also when we trigger the movement of
shards manually, it only moves that shard and nothing else. Doesn't cause
a big rebalance. It seems to still balance 2 out of the 5 nodes with
primaries while the other two have only replicas.
Is there anything we're missing?
Thanks!
On Wednesday, July 23, 2014 9:30:00 AM UTC-4, Nikolas Everett wrote:
Thanks for that, Nik. I'm okay with evenly spreading all the indices,
rather than just the one I'm having issues with. I'll give your config a
try!
Def no special configurations on that one. We didn't even realize there
was such a thing as allocation configuration up until yesterday (after the
0/6 allocation happened). The 0/6 node did, however, contain 4/6
replicas... If that makes a difference?
It certainly does. The way the allocation settings are configured now the
cluster thinks of replicas as just about the same as masters. For the most
part they are, too. For us they certainly are.
If you are sure that being a shard master is significantly more work then
being a shard replica then raise the "primary" setting to something like .5
and lower everything else to total 1.
That whole totally 1 thing isn't required but it makes the numbers easier
to think about.
I don't imagine there is anything you are missing. I don't know where to go
from here. Sorry!
On Jul 24, 2014 3:20 PM, "Daniel Schonfeld" downwindabeam@gmail.com wrote:
And it doesn't seem to do much... also when we trigger the movement of
shards manually, it only moves that shard and nothing else. Doesn't cause
a big rebalance. It seems to still balance 2 out of the 5 nodes with
primaries while the other two have only replicas.
Is there anything we're missing?
Thanks!
On Wednesday, July 23, 2014 9:30:00 AM UTC-4, Nikolas Everett wrote:
Thanks for that, Nik. I'm okay with evenly spreading all the indices,
rather than just the one I'm having issues with. I'll give your config a
try!
Def no special configurations on that one. We didn't even realize there
was such a thing as allocation configuration up until yesterday (after the
0/6 allocation happened). The 0/6 node did, however, contain 4/6
replicas... If that makes a difference?
It certainly does. The way the allocation settings are configured now
the cluster thinks of replicas as just about the same as masters. For the
most part they are, too. For us they certainly are.
If you are sure that being a shard master is significantly more work then
being a shard replica then raise the "primary" setting to something like .5
and lower everything else to total 1.
That whole totally 1 thing isn't required but it makes the numbers easier
to think about.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.