Hey All,
We're on ES 1.4.3. Initially, I thought I had this issue, however it
appears I was incorrect.
We're seeing very uneven primary shard distribution as we're adding new
indexes to our system. We're running a 6 node cluster. We're adding
indexes with 12 shards, and 1 replica. What I'm noticing is that 1/4 (3)
of the primary shards are be allocated to a single node. In the case of
our application, this is causing severe performance problems since we're
writing 1/3 of our throughput to a single node, leaving the remaining 2/3
spread over 5 nodes. We currently have 87 indexes and growing. I never
noticed this with a handful of indexes, but as we're getting more it's
becoming more prevalent.
What can I do to diagnose and fix this issue? From the screenshot above,
I've tried moving the primary shards, but this seems to have little effect.
When catting the shards and checking how many primaries exist per node,
they're very skewed
curl -X GET res001sy:9200/_cat/shards|grep "p STARTED"| grep res001sy|wc -l
=> 259
curl -X GET res001sy:9200/_cat/shards|grep "p STARTED"| grep res002sy|wc -l
=> 228
curl -X GET res001sy:9200/_cat/shards|grep "p STARTED"| grep res003sy|wc -l
=> 191
curl -X GET res001sy:9200/_cat/shards|grep "p STARTED"| grep res004sy|wc -l
=> 106
curl -X GET res001sy:9200/_cat/shards|grep "p STARTED"| grep res005sy|wc -l
=> 37
curl -X GET res001sy:9200/_cat/shards|grep "p STARTED"| grep res006sy|wc -l
=> 13
As you can see, 01 is our highest with 259 primary shards, however 06 only
had 13.
I've tried forcing shards to be rebalance with the command below but it
seems to have no effect.
Also, note that we're running the aws cloud plugin.
https://github.com/elasticsearch/elasticsearch-cloud-aws.
curl -XPOST 'res002sy:9200/_cluster/reroute' -d '{}'
Any ideas what I can do to resolve this problem? The uneven distribution
seems to killing our more heavily nodes.since they're taking a majority of
the primary shard throughput. I'd prefer that most nodes have an
approximately equal amount of primaries and secondaries, since our indexes
are mixes of more write heavy or more read heavy. We seem to have very few
that are 50/50.
Thanks,
Todd
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/7c935128-e783-48cc-8134-3bc2de0c9aaa%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.