# Index Size: Elasticsearch 1.4 -\> 2.1 **URL:** \ **Category:** Elasticsearch\ **Created:** [December 3, 2015, 7:50am UTC](https://discuss.elastic.co/t/index-size-elasticsearch-1-4-2-1/36253 "2015-12-03T07:50:07Z")\ **Posts on this page:** 18\ **Page:** 1
**Author:** ![Alexander\_Ott](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alexander_ott/32/105905_2.png) [@Alexander\_Ott](https://discuss.elastic.co/u/Alexander_Ott)\ **Post date:** [December 3, 2015, 7:50am UTC](https://discuss.elastic.co/t/index-size-elasticsearch-1-4-2-1/36253/1 "2015-12-03T07:50:07Z")
We currently switched from ES 1.4 to ES 2.1. After creation of our indices with ES 2.1 the size of them are more than twice bigger than with ES 1.4. Any hints about this effect or has anybody made the same observation? ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [December 3, 2015, 8:01am UTC](https://discuss.elastic.co/t/index-size-elasticsearch-1-4-2-1/36253/2 "2015-12-03T08:01:30Z")
Probably because of doc\_values which are activated by default? ---
**Author:** ![Alexander\_Ott](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alexander_ott/32/105905_2.png) [@Alexander\_Ott](https://discuss.elastic.co/u/Alexander_Ott)\ **Post date:** [December 3, 2015, 8:27am UTC](https://discuss.elastic.co/t/index-size-elasticsearch-1-4-2-1/36253/3 "2015-12-03T08:27:01Z")
And in ES 1.4 doc\_values weren't activated by default? ---
**Author:** ![rusty](https://avatars.discourse-cdn.com/v4/letter/r/f17d59/32.png) [@rusty](https://discuss.elastic.co/u/rusty)\ **Post date:** [December 3, 2015, 8:27am UTC](https://discuss.elastic.co/t/index-size-elasticsearch-1-4-2-1/36253/4 "2015-12-03T08:27:06Z")
Look at [Elasticsearch 2.0 2.5X Disk Space](https://discuss.elastic.co/t/elasticsearch-2-0-2-5x-disk-space/34805) seems to be same problem. ---
**Author:** ![Alexander\_Ott](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alexander_ott/32/105905_2.png) [@Alexander\_Ott](https://discuss.elastic.co/u/Alexander_Ott)\ **Post date:** [December 3, 2015, 8:28am UTC](https://discuss.elastic.co/t/index-size-elasticsearch-1-4-2-1/36253/5 "2015-12-03T08:28:01Z")
Thanks. I will check this. ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [December 3, 2015, 8:55am UTC](https://discuss.elastic.co/t/index-size-elasticsearch-1-4-2-1/36253/6 "2015-12-03T08:55:14Z")
> [@Alexander\_Ott](#): > > And in ES 1.4 doc\_values weren't activated by default? No they were not. ---
**Author:** ![Alexander\_Ott](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alexander_ott/32/105905_2.png) [@Alexander\_Ott](https://discuss.elastic.co/u/Alexander_Ott)\ **Post date:** [December 7, 2015, 8:53am UTC](https://discuss.elastic.co/t/index-size-elasticsearch-1-4-2-1/36253/7 "2015-12-07T08:53:03Z")
Is it possible to disable doc\_values by default as it was in 1.x? Or do we need to disable it on every field in our mapping now? ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [December 7, 2015, 9:51am UTC](https://discuss.elastic.co/t/index-size-elasticsearch-1-4-2-1/36253/8 "2015-12-07T09:51:26Z")
So you have `not_analyzed` fields but you don't use them for sorting or aggregations? Yes you have to define this for every field but you can use [Dynamic Templates](https://www.elastic.co/guide/en/elasticsearch/reference/current/dynamic-templates.html). ---
**Author:** ![Alexander\_Ott](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alexander_ott/32/105905_2.png) [@Alexander\_Ott](https://discuss.elastic.co/u/Alexander_Ott)\ **Post date:** [December 7, 2015, 3:31pm UTC](https://discuss.elastic.co/t/index-size-elasticsearch-1-4-2-1/36253/9 "2015-12-07T15:31:57Z")
Now i changed all our fields to doc\_values: false. But it seems that it doesn't work... Index size with ES.1.4 = 69GB index size with 2.1 = 172GB. May you can have a look at our mapping? Is it possible to upload it here. Upload function only allowes jpg, jpeg, png, gif ☹ ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [December 7, 2015, 3:33pm UTC](https://discuss.elastic.co/t/index-size-elasticsearch-1-4-2-1/36253/10 "2015-12-07T15:33:31Z")
You can pretty format it and copy and paste here if small. If not, paste it on [gist.github.com](http://gist.github.com) ---
**Author:** ![Alexander\_Ott](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alexander_ott/32/105905_2.png) [@Alexander\_Ott](https://discuss.elastic.co/u/Alexander_Ott)\ **Post date:** [December 7, 2015, 3:36pm UTC](https://discuss.elastic.co/t/index-size-elasticsearch-1-4-2-1/36253/11 "2015-12-07T15:36:53Z")
Please have a look at [https://gist.github.com/anonymous/c0ab1a97d655322cde55](https://gist.github.com/anonymous/c0ab1a97d655322cde55) ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [December 7, 2015, 3:52pm UTC](https://discuss.elastic.co/t/index-size-elasticsearch-1-4-2-1/36253/12 "2015-12-07T15:52:01Z")
Is it the same mapping you have for your `1.4` version? The mapping looks good. Did you index exactly the same data? ---
**Author:** ![Alexander\_Ott](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alexander_ott/32/105905_2.png) [@Alexander\_Ott](https://discuss.elastic.co/u/Alexander_Ott)\ **Post date:** [December 8, 2015, 8:25am UTC](https://discuss.elastic.co/t/index-size-elasticsearch-1-4-2-1/36253/13 "2015-12-08T08:25:20Z")
The mapping is the same up to "doc\_values: false" and under "\_source": { "enabled": true, "compressed": true } i removed the "compressed": true cause it isn't supported anymore and compression is enabled by default. isn't it? The data is exactly the same data and this is what makes me wonder. It seems that the doc\_values: false configuration dosen't work.... ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [December 8, 2015, 8:44am UTC](https://discuss.elastic.co/t/index-size-elasticsearch-1-4-2-1/36253/14 "2015-12-08T08:44:08Z")
@jpountz Any idea about this? ---
**Author:** ![jpountz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jpountz/32/45836_2.png) [@jpountz](https://discuss.elastic.co/u/jpountz)\ **Post date:** [December 10, 2015, 6:20pm UTC](https://discuss.elastic.co/t/index-size-elasticsearch-1-4-2-1/36253/15 "2015-12-10T18:20:09Z")
Alexandre, can you break down the size of your data directory by file extension for both versions? (eg. how much disk space are using the .fdt files, .dvm files, .tim files, etc.) ---
**Author:** ![Alexander\_Ott](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alexander_ott/32/105905_2.png) [@Alexander\_Ott](https://discuss.elastic.co/u/Alexander_Ott)\ **Post date:** [December 14, 2015, 12:10pm UTC](https://discuss.elastic.co/t/index-size-elasticsearch-1-4-2-1/36253/16 "2015-12-14T12:10:17Z")
At [https://gist.github.com/anonymous/120f63fbad5939febd92](https://gist.github.com/anonymous/120f63fbad5939febd92) you can find 4 files. The first two files list the files and size inside the data directory of each es 1.4 node. The last two files list the files and size inside the data directory of each es 2.1 node. ---
**Author:** ![jpountz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jpountz/32/45836_2.png) [@jpountz](https://discuss.elastic.co/u/jpountz)\ **Post date:** [December 14, 2015, 2:33pm UTC](https://discuss.elastic.co/t/index-size-elasticsearch-1-4-2-1/36253/17 "2015-12-14T14:33:52Z")
The main problem seems to be due to the fact that you have sparse analyzed string fields, ie. fields that are only present in a minority of documents. Norms were entirely stored in memory up to 2.0 included, which could occasionally take a lot of memory. In 2.1 norms have been moved to disk in order to reduce the memory requirements of Elasticsearch. However, while the new encoding requires much less memory (no memory at all actually), it also requires more disk space in the case that fields are sparse. We can look into better compressing norms on sparse fields, but this is not something that would come for free, in particular performance would be affected, and this would take quite some time to be released. However on your end, you could look into modeling your documents in such a way that you have fewer sparse analyzed string fields. Another option is to disable norms on your string fields if you don't need scoring. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 5, 2017, 11:31pm UTC](https://discuss.elastic.co/t/index-size-elasticsearch-1-4-2-1/36253/18 "2017-07-05T23:31:17Z")