# Shard size **URL:** \ **Category:** Elasticsearch\ **Created:** [June 3, 2013, 12:31am UTC](https://discuss.elastic.co/t/shard-size/12229 "2013-06-03T00:31:29Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![pleitao](https://avatars.discourse-cdn.com/v4/letter/p/45deac/32.png) [@pleitao](https://discuss.elastic.co/u/pleitao)\ **Post date:** [June 3, 2013, 12:31am UTC](https://discuss.elastic.co/t/shard-size/12229/1 "2013-06-03T00:31:29Z")
I've been trying to optimize ES for logstash; threads, indexing memory, number of shards, compression, etc and I'm stuck on disk usage. As a test I indexed 4GB of IIS logs into a two shard index on a two node cluster. Each shard has one replica. I'm seeing that each shard is 4GB and each replica is 4GB (fine, that's a given, it's a replica). In really confused as to why each shard is 4GB. Shouldn't both shards equate to 4GB, not 8GB? At this point, 4GB of logs is equating to 16GB of used storage across two nodes. routing: { state: STARTED primary: true node: eF-H3zhSTI6piq\_f6ukjtA relocating\_node: null shard: 0 index: logstash-2013.05.27 } state: STARTED index: { size: 3.8gb size\_in\_bytes: 4157721500 }m routing: { state: STARTED primary: true node: eF-H3zhSTI6piq\_f6ukjtA relocating\_node: null shard: 1 index: logstash-2013.05.27 } state: STARTED index: { size: 3.9gb size\_in\_bytes: 4191907329 } -confused -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![shadyabhi](https://avatars.discourse-cdn.com/v4/letter/s/edb3f5/32.png) [@shadyabhi](https://discuss.elastic.co/u/shadyabhi)\ **Post date:** [June 3, 2013, 4:54am UTC](https://discuss.elastic.co/t/shard-size/12229/2 "2013-06-03T04:54:57Z")
Hi pleitao, You said that you indexed 4GB logs and disk space by indices was 16GB. As you have 1 replica, it means that in addition to the primary shards, you'll also have replicas. (Hence, your primary shards occupy 8GB and your replicas 8GB). The output that you've pasted shows that those two shards are primary, so they don't include the replica, hence 3.9GB + 3.9GB = 8GB which makes sense. Hope it helps. On Mon, Jun 3, 2013 at 6:01 AM, [pleitao@gmail.com](mailto:pleitao@gmail.com) wrote: > I've been trying to optimize ES for logstash; threads, indexing memory, > number of shards, compression, etc and I'm stuck on disk usage. As a test I > indexed 4GB of IIS logs into a two shard index on a two node cluster. Each > shard has one replica. I'm seeing that each shard is 4GB and each replica > is 4GB (fine, that's a given, it's a replica). > > In really confused as to why each shard is 4GB. Shouldn't both shards > equate to 4GB, not 8GB? At this point, 4GB of logs is equating to 16GB of > used storage across two nodes. > > routing: { > state: STARTED > primary: true > node: eF-H3zhSTI6piq\_f6ukjtA > relocating\_node: null > shard: 0 > index: logstash-2013.05.27 > } > state: STARTED > index: { > size: 3.8gb > size\_in\_bytes: 4157721500 > }m > > routing: { > state: STARTED > primary: true > node: eF-H3zhSTI6piq\_f6ukjtA > relocating\_node: null > shard: 1 > index: logstash-2013.05.27 > } > state: STARTED > index: { > size: 3.9gb > size\_in\_bytes: 4191907329 > } > > -confused > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- Regards, Abhijeet Rastogi (shadyabhi) [http://blog.abhijeetr.com](http://blog.abhijeetr.com) -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![pleitao](https://avatars.discourse-cdn.com/v4/letter/p/45deac/32.png) [@pleitao](https://discuss.elastic.co/u/pleitao)\ **Post date:** [June 3, 2013, 10:46am UTC](https://discuss.elastic.co/t/shard-size/12229/3 "2013-06-03T10:46:50Z")
Thanks Abhijeet, Shouldn't shard 0 and shard 1 (the primary shards) equate to 4GB? I thought data was spread out across all shards, not duplicated on all shards. Thanks. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![shadyabhi](https://avatars.discourse-cdn.com/v4/letter/s/edb3f5/32.png) [@shadyabhi](https://discuss.elastic.co/u/shadyabhi)\ **Post date:** [June 3, 2013, 12:06pm UTC](https://discuss.elastic.co/t/shard-size/12229/4 "2013-06-03T12:06:17Z")
As you said: At this point, 4GB of logs is equating to 16GB of used storage across two nodes. This perfectly makes sense. As you've 2 primary shards, 1 replica configuration, you actually have 4 shards. Now, as 4GB converts to 16GB, this 16Gb is actually 8GB + 8GB. (as we have one replica). Now, as you've 2 primary shards, each 8GB is actually 4GB + 4GB. I'm not where I don't make sense. Also, there is no duplication of data among primary shards. On Mon, Jun 3, 2013 at 4:16 PM, [pleitao@gmail.com](mailto:pleitao@gmail.com) wrote: > Thanks Abhijeet, > Shouldn't shard 0 and shard 1 (the primary shards) equate to 4GB? I > thought data was spread out across all shards, not duplicated on all shards. > > Thanks. > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- Regards, Abhijeet Rastogi (shadyabhi) [http://blog.abhijeetr.com](http://blog.abhijeetr.com) -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Ivan](https://avatars.discourse-cdn.com/v4/letter/i/df788c/32.png) [@Ivan](https://discuss.elastic.co/u/Ivan)\ **Post date:** [June 3, 2013, 10:56pm UTC](https://discuss.elastic.co/t/shard-size/12229/5 "2013-06-03T22:56:20Z")
I think what the OP is asking is why does 4GBs of raw data end up being 8GB (non-replicated). Besides the size of the raw data, you need to account for the size of the field indexes as well. You can reduce the size by setting non-queried/filtered fields to be not indexed and not storing the fields. Fields are not stored by default, but they indexed by default. Also be sure that numeric fields are indexed as numerics and not strings. Use smaller numeric types whenever possible (ints/shorts over longs, float over double), but the most efficient savings is not indexing string content. Another source of potential savings is to omit term frequencies and/or norms. Read some Lucene documentation to understand what they do. -- Ivan On Mon, Jun 3, 2013 at 5:06 AM, Abhijeet Rastogi [abhijeet.1989@gmail.com](mailto:abhijeet.1989@gmail.com)wrote: > As you said: At this point, 4GB of logs is equating to 16GB of used > storage across two nodes. > > This perfectly makes sense. As you've 2 primary shards, 1 replica > configuration, you actually have 4 shards. > > Now, as 4GB converts to 16GB, this 16Gb is actually 8GB + 8GB. (as we have > one replica). Now, as you've 2 primary shards, each 8GB is actually 4GB + > 4GB. > > I'm not where I don't make sense. Also, there is no duplication of data > among primary shards. > > On Mon, Jun 3, 2013 at 4:16 PM, [pleitao@gmail.com](mailto:pleitao@gmail.com) wrote: > > > Thanks Abhijeet, > > Shouldn't shard 0 and shard 1 (the primary shards) equate to 4GB? I > > thought data was spread out across all shards, not duplicated on all shards. > > > > Thanks. > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > -- > Regards, > Abhijeet Rastogi (shadyabhi) > [http://blog.abhijeetr.com](http://blog.abhijeetr.com) > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 2:33am UTC](https://discuss.elastic.co/t/shard-size/12229/6 "2017-07-06T02:33:11Z")