# Some interesting storage numbers for people interested **URL:** \ **Category:** Elasticsearch\ **Created:** [February 23, 2015, 8:10pm UTC](https://discuss.elastic.co/t/some-interesting-storage-numbers-for-people-interested/22338 "2015-02-23T20:10:53Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![javadevmtl](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/javadevmtl/32/45613_2.png) [@javadevmtl](https://discuss.elastic.co/u/javadevmtl)\ **Post date:** [February 23, 2015, 8:10pm UTC](https://discuss.elastic.co/t/some-interesting-storage-numbers-for-people-interested/22338/1 "2015-02-23T20:10:53Z")
I don't run a blog but I thought I would share some results with the community. Using Elasticsearch 1.4.3 I wanted to test the various ways we could save some storage on our ES index and here are some numbers Created 6 different indexes with the various mapping settings. Each index containing 4 types. Insert 100,000 documents per type so total 400,000 per index. Average document size 300-400 bytes. The values represent the total primary space taken by each index based on the different mapping settings. \_source: true = 45MB \_source: true, \_all: false = 34MB \_source: false = 30MB \_source: false, \_all: false = 18MB \_source: false, store: true (all fields) = 39.5MB \_source: false, store: true (all fields), \_all: false = 28.5MB As you can see the default \_source setting takes the most space, while disabling the \_source and \_all field saves the most space. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/423ea99b-b9f2-4551-bb0c-d0167ed52150%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/423ea99b-b9f2-4551-bb0c-d0167ed52150%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![jackpark](https://avatars.discourse-cdn.com/v4/letter/j/5f8ce5/32.png) [@jackpark](https://discuss.elastic.co/u/jackpark)\ **Post date:** [February 23, 2015, 8:27pm UTC](https://discuss.elastic.co/t/some-interesting-storage-numbers-for-people-interested/22338/2 "2015-02-23T20:27:02Z")
What is lost (the tradeoff) when \_source is disabled? What is lost when \_all is disabled? This is interesting! Thanks Jack On Mon, Feb 23, 2015 at 12:10 PM, John Smith [java.dev.mtl@gmail.com](mailto:java.dev.mtl@gmail.com) wrote: > I don't run a blog but I thought I would share some results with the > community. > > Using Elasticsearch 1.4.3 > > I wanted to test the various ways we could save some storage on our ES > index and here are some numbers > > Created 6 different indexes with the various mapping settings. > Each index containing 4 types. > Insert 100,000 documents per type so total 400,000 per index. > Average document size 300-400 bytes. > > The values represent the total primary space taken by each index based on > the different mapping settings. > > \_source: true = 45MB > \_source: true, \_all: false = 34MB > \_source: false = 30MB > \_source: false, \_all: false = 18MB > \_source: false, store: true (all fields) = 39.5MB > \_source: false, store: true (all fields), \_all: false = 28.5MB > > As you can see the default \_source setting takes the most space, while > disabling the \_source and \_all field saves the most space. > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/423ea99b-b9f2-4551-bb0c-d0167ed52150%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/423ea99b-b9f2-4551-bb0c-d0167ed52150%40googlegroups.com) > [https://groups.google.com/d/msgid/elasticsearch/423ea99b-b9f2-4551-bb0c-d0167ed52150%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/423ea99b-b9f2-4551-bb0c-d0167ed52150%40googlegroups.com?utm_medium=email&utm_source=footer) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAH6s0fwgm%3DTJqN7Vqu7v3yUxg7OKz20rrpRgSx6HpApBRzWgpw%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAH6s0fwgm%3DTJqN7Vqu7v3yUxg7OKz20rrpRgSx6HpApBRzWgpw%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [February 23, 2015, 8:54pm UTC](https://discuss.elastic.co/t/some-interesting-storage-numbers-for-people-interested/22338/3 "2015-02-23T20:54:55Z")
Thanks John, this is a really interesting test. If you have no \_source you cannot reindex or view the actual raw content that was sent to ES, only the analysed portions you keep. No \_all means you have to know the exact field you want to search on or else you may get no results, as ES will search \_all by default (think of it as a shortcut search field). As an aside, we are working on adding a new compression algorithm for ES which will also improve storage capacity. On 24 February 2015 at 07:27, Jack Park [jackpark@topicquests.org](mailto:jackpark@topicquests.org) wrote: > What is lost (the tradeoff) when \_source is disabled? > What is lost when \_all is disabled? > > This is interesting! > > Thanks > Jack > > On Mon, Feb 23, 2015 at 12:10 PM, John Smith [java.dev.mtl@gmail.com](mailto:java.dev.mtl@gmail.com) > wrote: > > > I don't run a blog but I thought I would share some results with the > > community. > > > > Using Elasticsearch 1.4.3 > > > > I wanted to test the various ways we could save some storage on our ES > > index and here are some numbers > > > > Created 6 different indexes with the various mapping settings. > > Each index containing 4 types. > > Insert 100,000 documents per type so total 400,000 per index. > > Average document size 300-400 bytes. > > > > The values represent the total primary space taken by each index based on > > the different mapping settings. > > > > \_source: true = 45MB > > \_source: true, \_all: false = 34MB > > \_source: false = 30MB > > \_source: false, \_all: false = 18MB > > \_source: false, store: true (all fields) = 39.5MB > > \_source: false, store: true (all fields), \_all: false = 28.5MB > > > > As you can see the default \_source setting takes the most space, while > > disabling the \_source and \_all field saves the most space. > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/423ea99b-b9f2-4551-bb0c-d0167ed52150%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/423ea99b-b9f2-4551-bb0c-d0167ed52150%40googlegroups.com) > > [https://groups.google.com/d/msgid/elasticsearch/423ea99b-b9f2-4551-bb0c-d0167ed52150%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/423ea99b-b9f2-4551-bb0c-d0167ed52150%40googlegroups.com?utm_medium=email&utm_source=footer) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/CAH6s0fwgm%3DTJqN7Vqu7v3yUxg7OKz20rrpRgSx6HpApBRzWgpw%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAH6s0fwgm%3DTJqN7Vqu7v3yUxg7OKz20rrpRgSx6HpApBRzWgpw%40mail.gmail.com) > [https://groups.google.com/d/msgid/elasticsearch/CAH6s0fwgm%3DTJqN7Vqu7v3yUxg7OKz20rrpRgSx6HpApBRzWgpw%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAH6s0fwgm%3DTJqN7Vqu7v3yUxg7OKz20rrpRgSx6HpApBRzWgpw%40mail.gmail.com?utm_medium=email&utm_source=footer) > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAEYi1X-JaJOZ9bB7j62K1Y74QNGPZViYpNtpJWuU3nh-myUFfw%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEYi1X-JaJOZ9bB7j62K1Y74QNGPZViYpNtpJWuU3nh-myUFfw%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![jackpark](https://avatars.discourse-cdn.com/v4/letter/j/5f8ce5/32.png) [@jackpark](https://discuss.elastic.co/u/jackpark)\ **Post date:** [February 23, 2015, 10:04pm UTC](https://discuss.elastic.co/t/some-interesting-storage-numbers-for-people-interested/22338/4 "2015-02-23T22:04:29Z")
Thank you very much, Mark. On Mon, Feb 23, 2015 at 12:54 PM, Mark Walkom [markwalkom@gmail.com](mailto:markwalkom@gmail.com) wrote: > Thanks John, this is a really interesting test. > > If you have no \_source you cannot reindex or view the actual raw content > that was sent to ES, only the analysed portions you keep. > No \_all means you have to know the exact field you want to search on or > else you may get no results, as ES will search \_all by default (think of it > as a shortcut search field). > > As an aside, we are working on adding a new compression algorithm for ES > which will also improve storage capacity. > > On 24 February 2015 at 07:27, Jack Park [jackpark@topicquests.org](mailto:jackpark@topicquests.org) wrote: > > > What is lost (the tradeoff) when \_source is disabled? > > What is lost when \_all is disabled? > > > > This is interesting! > > > > Thanks > > Jack > > > > On Mon, Feb 23, 2015 at 12:10 PM, John Smith [java.dev.mtl@gmail.com](mailto:java.dev.mtl@gmail.com) > > wrote: > > > > > I don't run a blog but I thought I would share some results with the > > > community. > > > > > > Using Elasticsearch 1.4.3 > > > > > > I wanted to test the various ways we could save some storage on our ES > > > index and here are some numbers > > > > > > Created 6 different indexes with the various mapping settings. > > > Each index containing 4 types. > > > Insert 100,000 documents per type so total 400,000 per index. > > > Average document size 300-400 bytes. > > > > > > The values represent the total primary space taken by each index based > > > on the different mapping settings. > > > > > > \_source: true = 45MB > > > \_source: true, \_all: false = 34MB > > > \_source: false = 30MB > > > \_source: false, \_all: false = 18MB > > > \_source: false, store: true (all fields) = 39.5MB > > > \_source: false, store: true (all fields), \_all: false = 28.5MB > > > > > > As you can see the default \_source setting takes the most space, while > > > disabling the \_source and \_all field saves the most space. > > > > > > -- > > > You received this message because you are subscribed to the Google > > > Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send > > > an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > To view this discussion on the web visit > > > [https://groups.google.com/d/msgid/elasticsearch/423ea99b-b9f2-4551-bb0c-d0167ed52150%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/423ea99b-b9f2-4551-bb0c-d0167ed52150%40googlegroups.com) > > > [https://groups.google.com/d/msgid/elasticsearch/423ea99b-b9f2-4551-bb0c-d0167ed52150%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/423ea99b-b9f2-4551-bb0c-d0167ed52150%40googlegroups.com?utm_medium=email&utm_source=footer) > > > . > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/CAH6s0fwgm%3DTJqN7Vqu7v3yUxg7OKz20rrpRgSx6HpApBRzWgpw%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAH6s0fwgm%3DTJqN7Vqu7v3yUxg7OKz20rrpRgSx6HpApBRzWgpw%40mail.gmail.com) > > [https://groups.google.com/d/msgid/elasticsearch/CAH6s0fwgm%3DTJqN7Vqu7v3yUxg7OKz20rrpRgSx6HpApBRzWgpw%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAH6s0fwgm%3DTJqN7Vqu7v3yUxg7OKz20rrpRgSx6HpApBRzWgpw%40mail.gmail.com?utm_medium=email&utm_source=footer) > > . > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/CAEYi1X-JaJOZ9bB7j62K1Y74QNGPZViYpNtpJWuU3nh-myUFfw%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEYi1X-JaJOZ9bB7j62K1Y74QNGPZViYpNtpJWuU3nh-myUFfw%40mail.gmail.com) > [https://groups.google.com/d/msgid/elasticsearch/CAEYi1X-JaJOZ9bB7j62K1Y74QNGPZViYpNtpJWuU3nh-myUFfw%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAEYi1X-JaJOZ9bB7j62K1Y74QNGPZViYpNtpJWuU3nh-myUFfw%40mail.gmail.com?utm_medium=email&utm_source=footer) > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAH6s0fw6%2BE17PEBhq%3DMH7YeHME6tzEqgofwZ7Qnx\_BmQf3afkg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAH6s0fw6%2BE17PEBhq%3DMH7YeHME6tzEqgofwZ7Qnx_BmQf3afkg%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![javadevmtl](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/javadevmtl/32/45613_2.png) [@javadevmtl](https://discuss.elastic.co/u/javadevmtl)\ **Post date:** [February 23, 2015, 10:19pm UTC](https://discuss.elastic.co/t/some-interesting-storage-numbers-for-people-interested/22338/5 "2015-02-23T22:19:49Z")
Yeah sorry should have mentioned the tradeoffs. I think there some interesting use-cases here for instance if you are building a pure analytics dashboard where it's 100% aggregations then you can save allot of space with \_source: false, \_all: false In my case I'm opting for \_source: true, \_all: false. Since I need to re-index a document but don't care about the all search. My users are required to specify the field they want to search by specifying the field through a drop down... So it's good for the 25% saving On Monday, 23 February 2015 17:04:35 UTC-5, Jack Park wrote: > Thank you very much, Mark. > > On Mon, Feb 23, 2015 at 12:54 PM, Mark Walkom \<[markw...@gmail.com](mailto:markw...@gmail.com) > \\> wrote: > > > Thanks John, this is a really interesting test. > > > > If you have no \_source you cannot reindex or view the actual raw content > > that was sent to ES, only the analysed portions you keep. > > No \_all means you have to know the exact field you want to search on or > > else you may get no results, as ES will search \_all by default (think of it > > as a shortcut search field). > > > > As an aside, we are working on adding a new compression algorithm for ES > > which will also improve storage capacity. > > > > On 24 February 2015 at 07:27, Jack Park \<[jack...@topicquests.org](mailto:jack...@topicquests.org) > > \\> wrote: > > > > > What is lost (the tradeoff) when \_source is disabled? > > > What is lost when \_all is disabled? > > > > > > This is interesting! > > > > > > Thanks > > > Jack > > > > > > On Mon, Feb 23, 2015 at 12:10 PM, John Smith \<[java.d...@gmail.com](mailto:java.d...@gmail.com) > > > \\> wrote: > > > > > > > I don't run a blog but I thought I would share some results with the > > > > community. > > > > > > > > Using Elasticsearch 1.4.3 > > > > > > > > I wanted to test the various ways we could save some storage on our ES > > > > index and here are some numbers > > > > > > > > Created 6 different indexes with the various mapping settings. > > > > Each index containing 4 types. > > > > Insert 100,000 documents per type so total 400,000 per index. > > > > Average document size 300-400 bytes. > > > > > > > > The values represent the total primary space taken by each index based > > > > on the different mapping settings. > > > > > > > > \_source: true = 45MB > > > > \_source: true, \_all: false = 34MB > > > > \_source: false = 30MB > > > > \_source: false, \_all: false = 18MB > > > > \_source: false, store: true (all fields) = 39.5MB > > > > \_source: false, store: true (all fields), \_all: false = 28.5MB > > > > > > > > As you can see the default \_source setting takes the most space, while > > > > disabling the \_source and \_all field saves the most space. > > > > > > > > -- > > > > You received this message because you are subscribed to the Google > > > > Groups "elasticsearch" group. > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > > > To view this discussion on the web visit > > > > [https://groups.google.com/d/msgid/elasticsearch/423ea99b-b9f2-4551-bb0c-d0167ed52150%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/423ea99b-b9f2-4551-bb0c-d0167ed52150%40googlegroups.com) > > > > [https://groups.google.com/d/msgid/elasticsearch/423ea99b-b9f2-4551-bb0c-d0167ed52150%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/423ea99b-b9f2-4551-bb0c-d0167ed52150%40googlegroups.com?utm_medium=email&utm_source=footer) > > > > . > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > -- > > > You received this message because you are subscribed to the Google > > > Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > > To view this discussion on the web visit > > > [https://groups.google.com/d/msgid/elasticsearch/CAH6s0fwgm%3DTJqN7Vqu7v3yUxg7OKz20rrpRgSx6HpApBRzWgpw%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAH6s0fwgm%3DTJqN7Vqu7v3yUxg7OKz20rrpRgSx6HpApBRzWgpw%40mail.gmail.com) > > > [https://groups.google.com/d/msgid/elasticsearch/CAH6s0fwgm%3DTJqN7Vqu7v3yUxg7OKz20rrpRgSx6HpApBRzWgpw%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAH6s0fwgm%3DTJqN7Vqu7v3yUxg7OKz20rrpRgSx6HpApBRzWgpw%40mail.gmail.com?utm_medium=email&utm_source=footer) > > > . > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/CAEYi1X-JaJOZ9bB7j62K1Y74QNGPZViYpNtpJWuU3nh-myUFfw%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEYi1X-JaJOZ9bB7j62K1Y74QNGPZViYpNtpJWuU3nh-myUFfw%40mail.gmail.com) > > [https://groups.google.com/d/msgid/elasticsearch/CAEYi1X-JaJOZ9bB7j62K1Y74QNGPZViYpNtpJWuU3nh-myUFfw%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAEYi1X-JaJOZ9bB7j62K1Y74QNGPZViYpNtpJWuU3nh-myUFfw%40mail.gmail.com?utm_medium=email&utm_source=footer) > > . > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/36fb0b2f-3b80-4279-8f96-efba421cab51%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/36fb0b2f-3b80-4279-8f96-efba421cab51%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![javadevmtl](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/javadevmtl/32/45613_2.png) [@javadevmtl](https://discuss.elastic.co/u/javadevmtl)\ **Post date:** [February 23, 2015, 10:33pm UTC](https://discuss.elastic.co/t/some-interesting-storage-numbers-for-people-interested/22338/6 "2015-02-23T22:33:02Z")
Mark when you say you cannot re-index the document you mean re-index within the cluster? But if we resubmit the document using the index API it will get re-indexed and updated version 2 right? So Elastic search will mark the document to be deleted from the segment and eventually merge the "updated" data? On Monday, 23 February 2015 17:19:49 UTC-5, John Smith wrote: > Yeah sorry should have mentioned the tradeoffs. > > I think there some interesting use-cases here for instance if you are > building a pure analytics dashboard where it's 100% aggregations then you > can save allot of space with \_source: false, \_all: false > > In my case I'm opting for \_source: true, \_all: false. Since I need to > re-index a document but don't care about the all search. My users are > required to specify the field they want to search by specifying the field > through a drop down... So it's good for the 25% saving > > On Monday, 23 February 2015 17:04:35 UTC-5, Jack Park wrote: > > > Thank you very much, Mark. > > > > On Mon, Feb 23, 2015 at 12:54 PM, Mark Walkom [markw...@gmail.com](mailto:markw...@gmail.com) wrote: > > > > > Thanks John, this is a really interesting test. > > > > > > If you have no \_source you cannot reindex or view the actual raw content > > > that was sent to ES, only the analysed portions you keep. > > > No \_all means you have to know the exact field you want to search on or > > > else you may get no results, as ES will search \_all by default (think of it > > > as a shortcut search field). > > > > > > As an aside, we are working on adding a new compression algorithm for ES > > > which will also improve storage capacity. > > > > > > On 24 February 2015 at 07:27, Jack Park [jack...@topicquests.org](mailto:jack...@topicquests.org) wrote: > > > > > > > What is lost (the tradeoff) when \_source is disabled? > > > > What is lost when \_all is disabled? > > > > > > > > This is interesting! > > > > > > > > Thanks > > > > Jack > > > > > > > > On Mon, Feb 23, 2015 at 12:10 PM, John Smith [java.d...@gmail.com](mailto:java.d...@gmail.com) > > > > wrote: > > > > > > > > > I don't run a blog but I thought I would share some results with the > > > > > community. > > > > > > > > > > Using Elasticsearch 1.4.3 > > > > > > > > > > I wanted to test the various ways we could save some storage on our ES > > > > > index and here are some numbers > > > > > > > > > > Created 6 different indexes with the various mapping settings. > > > > > Each index containing 4 types. > > > > > Insert 100,000 documents per type so total 400,000 per index. > > > > > Average document size 300-400 bytes. > > > > > > > > > > The values represent the total primary space taken by each index based > > > > > on the different mapping settings. > > > > > > > > > > \_source: true = 45MB > > > > > \_source: true, \_all: false = 34MB > > > > > \_source: false = 30MB > > > > > \_source: false, \_all: false = 18MB > > > > > \_source: false, store: true (all fields) = 39.5MB > > > > > \_source: false, store: true (all fields), \_all: false = 28.5MB > > > > > > > > > > As you can see the default \_source setting takes the most space, while > > > > > disabling the \_source and \_all field saves the most space. > > > > > > > > > > -- > > > > > You received this message because you are subscribed to the Google > > > > > Groups "elasticsearch" group. > > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > > To view this discussion on the web visit > > > > > [https://groups.google.com/d/msgid/elasticsearch/423ea99b-b9f2-4551-bb0c-d0167ed52150%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/423ea99b-b9f2-4551-bb0c-d0167ed52150%40googlegroups.com) > > > > > [https://groups.google.com/d/msgid/elasticsearch/423ea99b-b9f2-4551-bb0c-d0167ed52150%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/423ea99b-b9f2-4551-bb0c-d0167ed52150%40googlegroups.com?utm_medium=email&utm_source=footer) > > > > > . > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > > > -- > > > > You received this message because you are subscribed to the Google > > > > Groups "elasticsearch" group. > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > To view this discussion on the web visit > > > > [https://groups.google.com/d/msgid/elasticsearch/CAH6s0fwgm%3DTJqN7Vqu7v3yUxg7OKz20rrpRgSx6HpApBRzWgpw%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAH6s0fwgm%3DTJqN7Vqu7v3yUxg7OKz20rrpRgSx6HpApBRzWgpw%40mail.gmail.com) > > > > [https://groups.google.com/d/msgid/elasticsearch/CAH6s0fwgm%3DTJqN7Vqu7v3yUxg7OKz20rrpRgSx6HpApBRzWgpw%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAH6s0fwgm%3DTJqN7Vqu7v3yUxg7OKz20rrpRgSx6HpApBRzWgpw%40mail.gmail.com?utm_medium=email&utm_source=footer) > > > > . > > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > -- > > > You received this message because you are subscribed to the Google > > > Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > To view this discussion on the web visit > > > [https://groups.google.com/d/msgid/elasticsearch/CAEYi1X-JaJOZ9bB7j62K1Y74QNGPZViYpNtpJWuU3nh-myUFfw%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEYi1X-JaJOZ9bB7j62K1Y74QNGPZViYpNtpJWuU3nh-myUFfw%40mail.gmail.com) > > > [https://groups.google.com/d/msgid/elasticsearch/CAEYi1X-JaJOZ9bB7j62K1Y74QNGPZViYpNtpJWuU3nh-myUFfw%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAEYi1X-JaJOZ9bB7j62K1Y74QNGPZViYpNtpJWuU3nh-myUFfw%40mail.gmail.com?utm_medium=email&utm_source=footer) > > > . > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/c2ab6c36-6199-48aa-88b9-ef007dc204c7%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/c2ab6c36-6199-48aa-88b9-ef007dc204c7%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [February 23, 2015, 10:39pm UTC](https://discuss.elastic.co/t/some-interesting-storage-numbers-for-people-interested/22338/7 "2015-02-23T22:39:28Z")
Yes, sorry. You can definitely reindex from an external source 🙂 On 24 February 2015 at 09:33, John Smith [java.dev.mtl@gmail.com](mailto:java.dev.mtl@gmail.com) wrote: > Mark when you say you cannot re-index the document you mean re-index > within the cluster? But if we resubmit the document using the index API it > will get re-indexed and updated version 2 right? > > So Elastic search will mark the document to be deleted from the segment > and eventually merge the "updated" data? > > On Monday, 23 February 2015 17:19:49 UTC-5, John Smith wrote: > > > Yeah sorry should have mentioned the tradeoffs. > > > > I think there some interesting use-cases here for instance if you are > > building a pure analytics dashboard where it's 100% aggregations then you > > can save allot of space with \_source: false, \_all: false > > > > In my case I'm opting for \_source: true, \_all: false. Since I need to > > re-index a document but don't care about the all search. My users are > > required to specify the field they want to search by specifying the field > > through a drop down... So it's good for the 25% saving > > > > On Monday, 23 February 2015 17:04:35 UTC-5, Jack Park wrote: > > > > > Thank you very much, Mark. > > > > > > On Mon, Feb 23, 2015 at 12:54 PM, Mark Walkom [markw...@gmail.com](mailto:markw...@gmail.com) > > > wrote: > > > > > > > Thanks John, this is a really interesting test. > > > > > > > > If you have no \_source you cannot reindex or view the actual raw > > > > content that was sent to ES, only the analysed portions you keep. > > > > No \_all means you have to know the exact field you want to search on or > > > > else you may get no results, as ES will search \_all by default (think of it > > > > as a shortcut search field). > > > > > > > > As an aside, we are working on adding a new compression algorithm for > > > > ES which will also improve storage capacity. > > > > > > > > On 24 February 2015 at 07:27, Jack Park [jack...@topicquests.org](mailto:jack...@topicquests.org) > > > > wrote: > > > > > > > > > What is lost (the tradeoff) when \_source is disabled? > > > > > What is lost when \_all is disabled? > > > > > > > > > > This is interesting! > > > > > > > > > > Thanks > > > > > Jack > > > > > > > > > > On Mon, Feb 23, 2015 at 12:10 PM, John Smith [java.d...@gmail.com](mailto:java.d...@gmail.com) > > > > > wrote: > > > > > > > > > > > I don't run a blog but I thought I would share some results with the > > > > > > community. > > > > > > > > > > > > Using Elasticsearch 1.4.3 > > > > > > > > > > > > I wanted to test the various ways we could save some storage on our > > > > > > ES index and here are some numbers > > > > > > > > > > > > Created 6 different indexes with the various mapping settings. > > > > > > Each index containing 4 types. > > > > > > Insert 100,000 documents per type so total 400,000 per index. > > > > > > Average document size 300-400 bytes. > > > > > > > > > > > > The values represent the total primary space taken by each index > > > > > > based on the different mapping settings. > > > > > > > > > > > > \_source: true = 45MB > > > > > > \_source: true, \_all: false = 34MB > > > > > > \_source: false = 30MB > > > > > > \_source: false, \_all: false = 18MB > > > > > > \_source: false, store: true (all fields) = 39.5MB > > > > > > \_source: false, store: true (all fields), \_all: false = 28.5MB > > > > > > > > > > > > As you can see the default \_source setting takes the most space, > > > > > > while disabling the \_source and \_all field saves the most space. > > > > > > > > > > > > -- > > > > > > You received this message because you are subscribed to the Google > > > > > > Groups "elasticsearch" group. > > > > > > To unsubscribe from this group and stop receiving emails from it, > > > > > > send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/) > > > > > > msgid/elasticsearch/423ea99b-b9f2-4551-bb0c-d0167ed52150% > > > > > > [40googlegroups.com](http://40googlegroups.com) > > > > > > [https://groups.google.com/d/msgid/elasticsearch/423ea99b-b9f2-4551-bb0c-d0167ed52150%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/423ea99b-b9f2-4551-bb0c-d0167ed52150%40googlegroups.com?utm_medium=email&utm_source=footer) > > > > > > . > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > > > > > -- > > > > > You received this message because you are subscribed to the Google > > > > > Groups "elasticsearch" group. > > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/) > > > > > msgid/elasticsearch/CAH6s0fwgm%3DTJqN7Vqu7v3yUxg7OKz20rrpRgSx > > > > > 6HpApBRzWgpw%[40mail.gmail.com](http://40mail.gmail.com) > > > > > [https://groups.google.com/d/msgid/elasticsearch/CAH6s0fwgm%3DTJqN7Vqu7v3yUxg7OKz20rrpRgSx6HpApBRzWgpw%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAH6s0fwgm%3DTJqN7Vqu7v3yUxg7OKz20rrpRgSx6HpApBRzWgpw%40mail.gmail.com?utm_medium=email&utm_source=footer) > > > > > . > > > > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > > > -- > > > > You received this message because you are subscribed to the Google > > > > Groups "elasticsearch" group. > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/) > > > > msgid/elasticsearch/CAEYi1X-JaJOZ9bB7j62K1Y74QNGPZViYpNtpJ > > > > WuU3nh-myUFfw%[40mail.gmail.com](http://40mail.gmail.com) > > > > [https://groups.google.com/d/msgid/elasticsearch/CAEYi1X-JaJOZ9bB7j62K1Y74QNGPZViYpNtpJWuU3nh-myUFfw%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAEYi1X-JaJOZ9bB7j62K1Y74QNGPZViYpNtpJWuU3nh-myUFfw%40mail.gmail.com?utm_medium=email&utm_source=footer) > > > > . > > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > -- > > > You received this message because you are subscribed to the Google Groups > > > "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send an > > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > To view this discussion on the web visit > > > [https://groups.google.com/d/msgid/elasticsearch/c2ab6c36-6199-48aa-88b9-ef007dc204c7%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/c2ab6c36-6199-48aa-88b9-ef007dc204c7%40googlegroups.com) > > > [https://groups.google.com/d/msgid/elasticsearch/c2ab6c36-6199-48aa-88b9-ef007dc204c7%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/c2ab6c36-6199-48aa-88b9-ef007dc204c7%40googlegroups.com?utm_medium=email&utm_source=footer) > > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAEYi1X-srv10LBFWBmjv%2BtO%2BisVLf1FkhkpdVX-9XRpXUrfj5w%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEYi1X-srv10LBFWBmjv%2BtO%2BisVLf1FkhkpdVX-9XRpXUrfj5w%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 12:30am UTC](https://discuss.elastic.co/t/some-interesting-storage-numbers-for-people-interested/22338/8 "2017-07-06T00:30:38Z")