# Uneven primary shard distribution with increasing index allocation

**URL:** <https://discuss.elastic.co/t/uneven-primary-shard-distribution-with-increasing-index-allocation/22224>\
**Category:** Elasticsearch\
**Created:** [February 18, 2015, 5:42am UTC](https://discuss.elastic.co/t/uneven-primary-shard-distribution-with-increasing-index-allocation/22224 "2015-02-18T05:42:50Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![Todd\_Nine](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/todd_nine/32/778_2.png) [@Todd\_Nine](https://discuss.elastic.co/u/Todd_Nine)\
**Post date:** [February 18, 2015, 5:42am UTC](https://discuss.elastic.co/t/uneven-primary-shard-distribution-with-increasing-index-allocation/22224/1 "2015-02-18T05:42:50Z")

</div>

Hey All,  
We're on ES 1.4.3. Initially, I thought I had this issue, however it  
appears I was incorrect.

> <https://github.com/elastic/elasticsearch/issues/9023>

We're seeing very uneven primary shard distribution as we're adding new  
indexes to our system. We're running a 6 node cluster. We're adding  
indexes with 12 shards, and 1 replica. What I'm noticing is that 1/4 (3)  
of the primary shards are be allocated to a single node. In the case of  
our application, this is causing severe performance problems since we're  
writing 1/3 of our throughput to a single node, leaving the remaining 2/3  
spread over 5 nodes. We currently have 87 indexes and growing. I never  
noticed this with a handful of indexes, but as we're getting more it's  
becoming more prevalent.

What can I do to diagnose and fix this issue? From the screenshot above,  
I've tried moving the primary shards, but this seems to have little effect.

When catting the shards and checking how many primaries exist per node,  
they're very skewed

curl -X GET res001sy:9200/\_cat/shards|grep "p STARTED"| grep res001sy|wc -l  
=\> 259  
curl -X GET res001sy:9200/\_cat/shards|grep "p STARTED"| grep res002sy|wc -l  
=\> 228  
curl -X GET res001sy:9200/\_cat/shards|grep "p STARTED"| grep res003sy|wc -l  
=\> 191  
curl -X GET res001sy:9200/\_cat/shards|grep "p STARTED"| grep res004sy|wc -l  
=\> 106  
curl -X GET res001sy:9200/\_cat/shards|grep "p STARTED"| grep res005sy|wc -l  
=\> 37  
curl -X GET res001sy:9200/\_cat/shards|grep "p STARTED"| grep res006sy|wc -l  
=\> 13

As you can see, 01 is our highest with 259 primary shards, however 06 only  
had 13.

I've tried forcing shards to be rebalance with the command below but it  
seems to have no effect.

Also, note that we're running the aws cloud plugin.  
[https://github.com/elasticsearch/elasticsearch-cloud-aws](https://github.com/elasticsearch/elasticsearch-cloud-aws).

curl -XPOST 'res002sy:9200/\_cluster/reroute' -d '{}'

Any ideas what I can do to resolve this problem? The uneven distribution  
seems to killing our more heavily nodes.since they're taking a majority of  
the primary shard throughput. I'd prefer that most nodes have an  
approximately equal amount of primaries and secondaries, since our indexes  
are mixes of more write heavy or more read heavy. We seem to have very few  
that are 50/50.

Thanks,  
Todd

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/7c935128-e783-48cc-8134-3bc2de0c9aaa%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/7c935128-e783-48cc-8134-3bc2de0c9aaa%40googlegroups.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![jnslndqst](https://avatars.discourse-cdn.com/v4/letter/j/4491bb/32.png) [@jnslndqst](https://discuss.elastic.co/u/jnslndqst)\
**Post date:** [May 24, 2016, 9:21am UTC](https://discuss.elastic.co/t/uneven-primary-shard-distribution-with-increasing-index-allocation/22224/2 "2016-05-24T09:21:36Z")

</div>

Todd, did you find any solution to this? I'm having the same experience on version 1.7.1

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 5, 2017, 10:49pm UTC](https://discuss.elastic.co/t/uneven-primary-shard-distribution-with-increasing-index-allocation/22224/3 "2017-07-05T22:49:36Z")

</div>