# Limit for shard size?

**URL:** https://discuss.elastic.co/t/limit-for-shard-size/40032
**Category:** Elasticsearch
**Created:** [January 25, 2016, 2:49pm UTC](https://discuss.elastic.co/t/limit-for-shard-size/40032 "2016-01-25T14:49:10Z")
**Posts on this page:** 3
**Page:** 1

<div class="post-metadata">

### Author: ![klahnakoski](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/klahnakoski/32/3679_2.png) [@klahnakoski](https://discuss.elastic.co/u/klahnakoski)
#### Post date: [January 25, 2016, 2:49pm UTC](https://discuss.elastic.co/t/limit-for-shard-size/40032/1 "2016-01-25T14:49:10Z")

</div>

My ES cluster has grown over the past few months. My drives are now 90% full, so I am adding more, while I am on the topic of managing-my-es-cluster I was wondering about maximum shard sizes. The biggest index is reporting 6billion documents, 4T bytes, over 24 shards, each shard approaching 200G.

The 200G shard size makes moving a shard a slow process. When I add a node, it can be over an hour of copying (cluster.routing.allocation.cluster\_concurrent\_rebalance: 1) before it can take over the query load. I believe smaller shards will replicate faster, allowing the new node to participate sooner. Am I right about this?

How small should the shards be? 20G shards will copy over in 1/10 the time (about 6min), but that would mean 240 shards! That sounds excessive! What are the memory requirements for a node (node.master=true, node.data=false) that must aggregate query results from so many shards?

On the subject of hard limits: I read that a Lucene index has a 2billion document limit. Good, I am far from hitting that. I also read that [ES version 2.0+ will put shards on one datapath](https://github.com/elastic/elasticsearch/issues/9498). Does that mean my datapaths must be larger than the 200G shards?

Thanks

---

<div class="post-metadata">

### Author: ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)
#### Post date: [January 26, 2016, 1:52am UTC](https://discuss.elastic.co/t/limit-for-shard-size/40032/2 "2016-01-26T01:52:19Z")

</div>

> [@klahnakoski](#):
>
> 24 shards, each shard approaching 200G

That's larger than we recommend. We suggest 50GB simply because moving more than that around due to (re)allocation takes a looooong time.

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 5, 2017, 11:21pm UTC](https://discuss.elastic.co/t/limit-for-shard-size/40032/3 "2017-07-05T23:21:47Z")

</div>


