# The shards are crowded in a node **URL:** \ **Category:** Elasticsearch\ **Created:** [December 19, 2017, 2:36am UTC](https://discuss.elastic.co/t/the-shards-are-crowded-in-a-node/112335 "2017-12-19T02:36:31Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![mcun0s](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mcun0s/32/25762_2.png) [@mcun0s](https://discuss.elastic.co/u/mcun0s)\ **Post date:** [December 19, 2017, 2:36am UTC](https://discuss.elastic.co/t/the-shards-are-crowded-in-a-node/112335/1 "2017-12-19T02:36:31Z")
Hi 🙂 We're using Elasticsearch for almost our products and we found a strange issue on our Elasticsearch cluster. The problem is that the shards are crowded in a node for a couple of indexes. **Issue Image** ![25487227_1479420818846070_2633026120816862982_o](https://us1.discourse-cdn.com/elastic/original/3X/a/9/a91456ce63eff6dfa6357d8fba84b2f3cb4758ce.jpg) Please check the above image. **Elasticsearch Information** - Elasticsearch version : 5.5 - OS version : CentOS 6.9 - 3 Master Nodes - 7 Hot Data Nodes - - Shards configuration : 24 - - Replica configuration : 0 **Changes** We've used that 5 data nodes and then recently, we've added 2 data nodes in the clusters. After then, the shards are crowded in a 6th node. **Reproduce Issue** I can't say that how to reproduce this issue because other indexes are looking good. Obviously, if we created new indexes then it could occur this issue. **Workaround** I believe we can resolve this issue if we can relocate the shards manually. But I worried about occur again after doing that. **Questions** So, my questions are, 1. Is this a reasonable issue if the additional data nodes are added in exists cluster? 2. Is Elasticsearch performance still good if the shards are crowded in a single node? 3. If this topic could be an issue, how do we fix it? Thanks for your help! Best regards, Unho. ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [December 19, 2017, 6:03am UTC](https://discuss.elastic.co/t/the-shards-are-crowded-in-a-node/112335/2 "2017-12-19T06:03:41Z")
Is the available disk space lower on node 1 than node 2? ---
**Author:** ![zqc0512](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/zqc0512/32/32141_2.png) [@zqc0512](https://discuss.elastic.co/u/zqc0512)\ **Post date:** [December 19, 2017, 6:25am UTC](https://discuss.elastic.co/t/the-shards-are-crowded-in-a-node/112335/3 "2017-12-19T06:25:47Z")
see the disk /rebanlce /shards about pre node. ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [December 19, 2017, 7:46am UTC](https://discuss.elastic.co/t/the-shards-are-crowded-in-a-node/112335/4 "2017-12-19T07:46:09Z")
Why do you have so many primary shards? ---
**Author:** ![mcun0s](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mcun0s/32/25762_2.png) [@mcun0s](https://discuss.elastic.co/u/mcun0s)\ **Post date:** [December 19, 2017, 8:51am UTC](https://discuss.elastic.co/t/the-shards-are-crowded-in-a-node/112335/5 "2017-12-19T08:51:25Z")
Hi David, thanks for your help. The crowded node is node6 and the disk size of node6 is lower than other. You can see the shard information like below. ```auto node shards disk.indices disk.percent hot-1 1487 581gb 76 hot-2 1487 548.1gb 77 hot-3 1487 557.2gb 43 hot-4 1487 557.8gb 78 hot-5 1487 554.9gb 80 hot-6 1416 462.9gb 89 hot-7 1487 571.5gb 76 ``` ---
**Author:** ![mcun0s](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mcun0s/32/25762_2.png) [@mcun0s](https://discuss.elastic.co/u/mcun0s)\ **Post date:** [December 19, 2017, 8:52am UTC](https://discuss.elastic.co/t/the-shards-are-crowded-in-a-node/112335/6 "2017-12-19T08:52:53Z")
The index size is too large. So we determined to split data using shard. ---
**Author:** ![mcun0s](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mcun0s/32/25762_2.png) [@mcun0s](https://discuss.elastic.co/u/mcun0s)\ **Post date:** [December 19, 2017, 8:59am UTC](https://discuss.elastic.co/t/the-shards-are-crowded-in-a-node/112335/7 "2017-12-19T08:59:33Z")
Could you see the below information? I'm not sure "the disk /rebalance" word. > node shards disk.indices disk.percent > hot-1 1487 581gb 76 > hot-2 1487 548.1gb 77 > hot-3 1487 557.2gb 43 > hot-4 1487 557.8gb 78 > hot-5 1487 554.9gb 80 > hot-6 1416 462.9gb 89 > hot-7 1487 571.5gb 76 ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [December 19, 2017, 9:04am UTC](https://discuss.elastic.co/t/the-shards-are-crowded-in-a-node/112335/8 "2017-12-19T09:04:34Z")
It's probably caused by [Disk-based shard allocation | Elasticsearch Guide [8.11] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/disk-allocator.html) ? > `cluster.routing.allocation.disk.watermark.low` > Controls the low watermark for disk usage. **It defaults to 85%** , meaning ES will not allocate new shards to nodes once they have more than 85% disk used. It can also be set to an absolute byte value (like 500mb) to prevent ES from allocating shards if less than the configured amount of space is available. ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [December 19, 2017, 9:08am UTC](https://discuss.elastic.co/t/the-shards-are-crowded-in-a-node/112335/9 "2017-12-19T09:08:59Z")
You have too many shards, you could easily double the size of the current shards. ---
**Author:** ![zqc0512](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/zqc0512/32/32141_2.png) [@zqc0512](https://discuss.elastic.co/u/zqc0512)\ **Post date:** [December 20, 2017, 12:33am UTC](https://discuss.elastic.co/t/the-shards-are-crowded-in-a-node/112335/10 "2017-12-20T00:33:21Z")
> [@mcun0s](#): > > node shards disk.indices disk.percent > > hot-1 1487 581gb 76 > > hot-2 1487 548.1gb 77 > > hot-3 1487 557.2gb 43 > > hot-4 1487 557.8gb 78 > > hot-5 1487 554.9gb 80 > > hot-6 1416 462.9gb 89 > > hot-7 1487 571.5gb 76 the disks space of per node not same? ---
**Author:** ![mcun0s](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mcun0s/32/25762_2.png) [@mcun0s](https://discuss.elastic.co/u/mcun0s)\ **Post date:** [December 21, 2017, 2:03am UTC](https://discuss.elastic.co/t/the-shards-are-crowded-in-a-node/112335/11 "2017-12-21T02:03:10Z")
Hm... If you theory is correct, then the 6th nodes should be not allocated new shards, but it's not. But it's useful information for me. I didn't know this option, so that would be usable future version. Thanks a lot! ---
**Author:** ![mcun0s](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mcun0s/32/25762_2.png) [@mcun0s](https://discuss.elastic.co/u/mcun0s)\ **Post date:** [December 21, 2017, 2:09am UTC](https://discuss.elastic.co/t/the-shards-are-crowded-in-a-node/112335/12 "2017-12-21T02:09:49Z")
Yeah, as you know that, we have too many shards. But disk capacity is increasing gradually because new documents are stored. We cannot expect how much data will be stored. Do you have any strategy on that? I think we can split each shards by data size, but ES doesn't support on that. So, we've determined that the shards size should be 24 ~ 40. Do you have any recommendation? Best regards, Unho. ---
**Author:** ![zqc0512](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/zqc0512/32/32141_2.png) [@zqc0512](https://discuss.elastic.co/u/zqc0512)\ **Post date:** [December 21, 2017, 2:58am UTC](https://discuss.elastic.co/t/the-shards-are-crowded-in-a-node/112335/13 "2017-12-21T02:58:04Z")
> [@mcun0s](#): > > Yeah, as you know that, we have too many shards. > > But disk capacity is increasing gradually because new documents are stored. as i know it support. see the docs. about elasticsearch. ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [December 21, 2017, 3:25am UTC](https://discuss.elastic.co/t/the-shards-are-crowded-in-a-node/112335/14 "2017-12-21T03:25:38Z")
> [@mcun0s](#): > > I think we can split each shards by data size, but ES doesn't support on that. It does in 6.1 - [Split Index | Elasticsearch Reference [6.1] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/6.1/indices-split-index.html) ---
**Author:** ![mcun0s](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mcun0s/32/25762_2.png) [@mcun0s](https://discuss.elastic.co/u/mcun0s)\ **Post date:** [December 21, 2017, 7:48am UTC](https://discuss.elastic.co/t/the-shards-are-crowded-in-a-node/112335/15 "2017-12-21T07:48:22Z")
I think the docs of as your mentioned are shared by Mark, right? I will review that and then share to my team. Thanks for all of your helps! Best regards, Unho. ---
**Author:** ![mcun0s](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mcun0s/32/25762_2.png) [@mcun0s](https://discuss.elastic.co/u/mcun0s)\ **Post date:** [December 21, 2017, 7:52am UTC](https://discuss.elastic.co/t/the-shards-are-crowded-in-a-node/112335/16 "2017-12-21T07:52:56Z")
Your information is so useful. Thank you so much, Mark! It seems like our ES has to be upgraded to 6.1 version.(We're using 5.5 ES version) We need to make a plan to upgrade to use that. I will share this document with my team! Have a great day! Best regards, Unho. ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [December 21, 2017, 8:18am UTC](https://discuss.elastic.co/t/the-shards-are-crowded-in-a-node/112335/17 "2017-12-21T08:18:18Z")
Adding also this presentation in case it helps: > **[NetSecureDay: Managing your Black Friday Logs](https://speakerdeck.com/elastic/netsecureday-managing-your-black-friday-logs)** > > Surveiller une application complexe n’est pas une tâche aisée, mais avec les bons outils, ce n’est pas si sorcier. Néanmoins, des périodes fortes telles que les opérations de type « Black Friday » (Vendredi noir) ou période de Noël peuvent pousser... ---
**Author:** ![zqc0512](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/zqc0512/32/32141_2.png) [@zqc0512](https://discuss.elastic.co/u/zqc0512)\ **Post date:** [December 21, 2017, 8:20am UTC](https://discuss.elastic.co/t/the-shards-are-crowded-in-a-node/112335/18 "2017-12-21T08:20:20Z")
with 5.5 as i know can rebuild index with mapping also can slove it update to 6.1 need so many test . [https://www.elastic.co/guide/en/elasticsearch/reference/5.5/indices-shrink-index.html](https://www.elastic.co/guide/en/elasticsearch/reference/5.5/indices-shrink-index.html) ---
**Author:** ![mcun0s](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mcun0s/32/25762_2.png) [@mcun0s](https://discuss.elastic.co/u/mcun0s)\ **Post date:** [December 22, 2017, 12:57am UTC](https://discuss.elastic.co/t/the-shards-are-crowded-in-a-node/112335/19 "2017-12-22T00:57:10Z")
Actually, we manage the indices by date. Even though, we made overshard. Thanks for sharing the document. I've learned good technics from you guys, I believe that would be nice to my team and me. Thanks again. ---
**Author:** ![mcun0s](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mcun0s/32/25762_2.png) [@mcun0s](https://discuss.elastic.co/u/mcun0s)\ **Post date:** [December 22, 2017, 12:58am UTC](https://discuss.elastic.co/t/the-shards-are-crowded-in-a-node/112335/20 "2017-12-22T00:58:31Z")
Yes, as you told me, we gonna upgrade carefully to our Elasticsearch, thanks for your advice! [Next page](https://discuss.elastic.co/t/the-shards-are-crowded-in-a-node/112335.md?page=2)