Hi All,
We have several indexes in our ES cluster. ES is not our canonical
system record, we use it only for searching.
Some of our applications have very high write throughput, so for these we
allocate a singular primary shard for each of our nodes. For example, we
have 6 nodes, and we create our index with 6 shards and 0 replicas. I
would expect ES to balance primary shards per node in the index. However,
it will load as many as 3 primaries on a single node, and some nodes will
have no shards. While each node does appear to have the same number of
primaries, not all our indexes have equal write throughput, therefore, I
want the primary (and only) shards evenly distributed per index.
During heavy writes, we experience slowness because our shards are not
evenly distributed across our nodes. Currently we're getting around this
by manually moving shards, but this seems to be something ES should do
automatically. We're running version 1.4.3 with default allocation
parameter. Is this a bug?
It's not a bug, ES allocates based on higher level primary shard count and
doesn't take into account what index a shard may belong to, this is where
allocation awareness and routing comes into play.
Hi All,
We have several indexes in our ES cluster. ES is not our canonical
system record, we use it only for searching.
Some of our applications have very high write throughput, so for these we
allocate a singular primary shard for each of our nodes. For example, we
have 6 nodes, and we create our index with 6 shards and 0 replicas. I
would expect ES to balance primary shards per node in the index. However,
it will load as many as 3 primaries on a single node, and some nodes will
have no shards. While each node does appear to have the same number of
primaries, not all our indexes have equal write throughput, therefore, I
want the primary (and only) shards evenly distributed per index.
During heavy writes, we experience slowness because our shards are not
evenly distributed across our nodes. Currently we're getting around this
by manually moving shards, but this seems to be something ES should do
automatically. We're running version 1.4.3 with default allocation
parameter. Is this a bug?
Thanks Mark,
I've looked at these, as well as these configuration parameters.
However, I've not been able to find a configuration that actually
accomplishes what we're looking for. We're using the aws plugin as well,
which doesn't seem to help any with our shard allocation. It does however,
ensure we have replica's in other zones, which is obviously helpful for
redundancy.
Are there any known settings, either with allocation-awareness or cluster
settings that will accomplish the behavior I'm looking for? Otherwise,
we'll have to write client API to move the shards after index allocation
to ensure we get our even load.
Thanks,
Todd
On Sunday, February 22, 2015 at 3:25:28 PM UTC-7, Mark Walkom wrote:
It's not a bug, ES allocates based on higher level primary shard count and
doesn't take into account what index a shard may belong to, this is where
allocation awareness and routing comes into play.
On 23 February 2015 at 06:46, Todd Nine <tn...@apigee.com <javascript:>>
wrote:
Hi All,
We have several indexes in our ES cluster. ES is not our canonical
system record, we use it only for searching.
Some of our applications have very high write throughput, so for these we
allocate a singular primary shard for each of our nodes. For example, we
have 6 nodes, and we create our index with 6 shards and 0 replicas. I
would expect ES to balance primary shards per node in the index. However,
it will load as many as 3 primaries on a single node, and some nodes will
have no shards. While each node does appear to have the same number of
primaries, not all our indexes have equal write throughput, therefore, I
want the primary (and only) shards evenly distributed per index.
During heavy writes, we experience slowness because our shards are not
evenly distributed across our nodes. Currently we're getting around this
by manually moving shards, but this seems to be something ES should do
automatically. We're running version 1.4.3 with default allocation
parameter. Is this a bug?
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.