# Index size seems massive to what the data is being sent **URL:** \ **Category:** Elasticsearch\ **Created:** [April 5, 2018, 5:43pm UTC](https://discuss.elastic.co/t/index-size-seems-massive-to-what-the-data-is-being-sent/126977 "2018-04-05T17:43:16Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![kmroz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kmroz/32/62741_2.png) [@kmroz](https://discuss.elastic.co/u/kmroz)\ **Post date:** [April 5, 2018, 5:43pm UTC](https://discuss.elastic.co/t/index-size-seems-massive-to-what-the-data-is-being-sent/126977/1 "2018-04-05T17:43:16Z")
Hello, I am sending metrics via collectd on a 10s interval rate. my daily index comes out to a size from 25-27gb. This seems a bit massive? The index settings are setup as follows "number\_of\_shards": "5", "number\_of\_replicas": "1". I have a total of 12KV that are sent in the message. Is there something else that can be making this index so large? ---
**Author:** ![Evesy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/evesy/32/29520_2.png) [@Evesy](https://discuss.elastic.co/u/Evesy)\ **Post date:** [April 6, 2018, 4:52am UTC](https://discuss.elastic.co/t/index-size-seems-massive-to-what-the-data-is-being-sent/126977/2 "2018-04-06T04:52:29Z")
Hey Kenneth, Elasticsearch does a number of things to make your data more easily searchable, all of which will add some overhead to your documents, and therefore index size -- On top of that you also have 1 replica, so the data volume will be doubled immediately. Analysed fields (or text fields in ES 6.x) will use some overhead since a field is split up into individual terms -- If you have any string fields that don't require full text search (i.e. you know the exact thing you want to search for), consider setting those fields `not_analyzed`, or `keyword` type in ES 6. You also have two additional special fields that will cause bloat in the size of your documents: `_source` -- The original raw JSON in its entirety. It can be disabled but there are some warnings/considerations to take first: [https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-source-field.html](https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-source-field.html) `_all` (Remove in 6.x) -- A concatenation of all the values of your fields, for when you want to search for a specific value, but you don't care about a particular field. Off the top of my head, not sure if this is enabled by default or not. All of the above will use additional storage overhead easily resulting in a document that's larger than the original document size, upto x2, x3 etc. You can tweak those settings in your index (and also not index any fields you don't care about) which could help save some space. You can also apply a difference codec to your index, i.e. `best_compression` which will help keep storage usage down, but will have some CPU overhead: [https://www.elastic.co/blog/store-compression-in-lucene-and-elasticsearch](https://www.elastic.co/blog/store-compression-in-lucene-and-elasticsearch) Hope that helps Cheers, Mike ---
**Author:** ![kmroz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kmroz/32/62741_2.png) [@kmroz](https://discuss.elastic.co/u/kmroz)\ **Post date:** [April 6, 2018, 2:39pm UTC](https://discuss.elastic.co/t/index-size-seems-massive-to-what-the-data-is-being-sent/126977/3 "2018-04-06T14:39:04Z")
I will give this a try. Thank you @Evesy ---
**Author:** ![kmroz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kmroz/32/62741_2.png) [@kmroz](https://discuss.elastic.co/u/kmroz)\ **Post date:** [April 16, 2018, 2:25pm UTC](https://discuss.elastic.co/t/index-size-seems-massive-to-what-the-data-is-being-sent/126977/4 "2018-04-16T14:25:23Z")
@Evesy we tried to disable the \_source option but that seems to break alot of stuff so that is not an option for us. I am still curious to how this data set is so large. We did a trail run on writing striaght to disk and it comes out to about a .5 gb per day. But ES seems to make it 10-20x that . Is there something I am missing in what is being done within ES? Thanks for any help!! ---
**Author:** ![Evesy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/evesy/32/29520_2.png) [@Evesy](https://discuss.elastic.co/u/Evesy)\ **Post date:** [April 16, 2018, 2:57pm UTC](https://discuss.elastic.co/t/index-size-seems-massive-to-what-the-data-is-being-sent/126977/5 "2018-04-16T14:57:08Z")
I'm sure someone with more knowledge can weigh in with more details around what causes inflated disk usage in Elastic compared to raw data storage. I've found [https://qbox.io/blog/elasticsearch-5-0-disk-usage-tuning](https://qbox.io/blog/elasticsearch-5-0-disk-usage-tuning) to be quite useful in reducing storage use though, there's quite a few options in there that are a lot less 'breaking' than disabling the `_source` field. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [May 14, 2018, 2:57pm UTC](https://discuss.elastic.co/t/index-size-seems-massive-to-what-the-data-is-being-sent/126977/6 "2018-05-14T14:57:34Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.