# How is data within an index distributed into shards?
**URL:** https://discuss.elastic.co/t/how-is-data-within-an-index-distributed-into-shards/121670
**Category:** Elasticsearch
**Created:** [February 27, 2018, 2:02pm UTC](https://discuss.elastic.co/t/how-is-data-within-an-index-distributed-into-shards/121670 "2018-02-27T14:02:46Z")
**Posts on this page:** 7
**Page:** 1
### Author:  [@Jan\_Pisl](https://discuss.elastic.co/u/Jan_Pisl)
#### Post date: [February 27, 2018, 2:02pm UTC](https://discuss.elastic.co/t/how-is-data-within-an-index-distributed-into-shards/121670/1 "2018-02-27T14:02:47Z")
I am new to ES and although I have read several blog posts and articles, I still cannot find the answers I am looking for. I need to ensure that shards in my ES database will have the optimal size (performance-wise).
I understand that the optimal size of a shard depends on multiple factors and that it (I suppose) can only be determined by tests with using real data.
What I don't understand is how data is distributed into shards within an index. For example, if there are five (primary) shards in an index and I set the maximal index size to 100 GB, will each shard be 20 GB when the index reaches its limit? So, if I find out the optimal shard size I can just multiply it by the number of shards in an index and use this value as the maximal index size? Or is it not this simple?
And more generally, why is it that there is no tool for simply specifying the maximal shard size instead of the maximal index size (such se the Rollover API) ?
---
### Author:  [@dadoonet](https://discuss.elastic.co/u/dadoonet)
#### Post date: [February 27, 2018, 2:16pm UTC](https://discuss.elastic.co/t/how-is-data-within-an-index-distributed-into-shards/121670/2 "2018-02-27T14:16:42Z")
> and I set the maximal index size to 100 GB
No you never do that with elasticsearch. Sky is the limit đ
May I suggest you look at the following resources about sizing:
> **[Quantitative Cluster Sizing](https://www.elastic.co/elasticon/conf/2016/sf/quantitative-cluster-sizing)**
> **[How many shards should I have in my Elasticsearch cluster?](https://www.elastic.co/blog/how-many-shards-should-i-have-in-my-elasticsearch-cluster)**
> **[NetSecureDay: Managing your Black Friday Logs](https://speakerdeck.com/elastic/netsecureday-managing-your-black-friday-logs)**
>
> Surveiller une application complexe nâest pas une tĂąche aisĂ©e, mais avec les bons outils, ce nâest pas si sorcier. NĂ©anmoins, des pĂ©riodes fortes telles que les opĂ©rations de type « Black Friday » (Vendredi noir) ou pĂ©riode de NoĂ«l peuvent pousser...
[ ](https://www.youtube.com/watch?v=ilP7tG6tabI)
---
### Author:  [@Jan\_Pisl](https://discuss.elastic.co/u/Jan_Pisl)
#### Post date: [February 27, 2018, 2:29pm UTC](https://discuss.elastic.co/t/how-is-data-within-an-index-distributed-into-shards/121670/3 "2018-02-27T14:29:21Z")
Thank you for you response!
I will most certainly go through the recourses you suggested. However, I am a bit perplexed by you saying
> [@dadoonet](#):
>
> No you never do that with elasticsearch. Sky is the limit
because in the very article you recommend me (and that I have already read) "How many shards should I have in my Elasticsearch cluster?", they say that "a shard size of 50GB is often quoted as a limit that has been seen to work". So doesn't it make sense to limit the size of indices then (because they consist of shards)? Or how else do I control the size of shards? I feel like I am missing something here...
---
### Author:  [@dadoonet](https://discuss.elastic.co/u/dadoonet)
#### Post date: [February 27, 2018, 2:46pm UTC](https://discuss.elastic.co/t/how-is-data-within-an-index-distributed-into-shards/121670/4 "2018-02-27T14:46:02Z")
Yes. That's a theorical limit. If you send more data, elasticsearch will still accept it and index it.
So there is no "limit" as an index settings or something like this.
I'd say that the only limit I know for now is the available disk space.
> So doesn't it make sense to limit the size of indices
yes. But elasticsearch does not have that built-in setting as I said.
> Or how else do I control the size of shards?
You can use the rollover API to achieve what you are looking for: keeping data under a given limit per shard.
See [Rollover Index | Elasticsearch Reference [6.2] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/6.2/indices-rollover-index.html)
---
### Author:  [@Jan\_Pisl](https://discuss.elastic.co/u/Jan_Pisl)
#### Post date: [February 27, 2018, 3:00pm UTC](https://discuss.elastic.co/t/how-is-data-within-an-index-distributed-into-shards/121670/5 "2018-02-27T15:00:44Z")
Ok so just to clarify I am on the same page: the way to control shard size (keep data under a given limit per shard) is to control index size & number of shards per index. do you agree on that?
thank you very much your answers are very helpful to me! â
---
### Author:  [@dadoonet](https://discuss.elastic.co/u/dadoonet)
#### Post date: [February 27, 2018, 6:35pm UTC](https://discuss.elastic.co/t/how-is-data-within-an-index-distributed-into-shards/121670/6 "2018-02-27T18:35:55Z")
I agree on that.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [March 27, 2018, 6:35pm UTC](https://discuss.elastic.co/t/how-is-data-within-an-index-distributed-into-shards/121670/7 "2018-03-27T18:35:57Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.