# Custom \_source compression / compaction to reduce disk usage **URL:** \ **Category:** Elasticsearch\ **Created:** [December 15, 2014, 2:20pm UTC](https://discuss.elastic.co/t/custom--source-compression-compaction-to-reduce-disk-usage/21258 "2014-12-15T14:20:14Z")\ **Posts on this page:** 14\ **Page:** 1
**Author:** ![Eran\_Duchan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/eran_duchan/32/1029_2.png) [@Eran\_Duchan](https://discuss.elastic.co/u/Eran_Duchan)\ **Post date:** [December 15, 2014, 2:20pm UTC](https://discuss.elastic.co/t/custom--source-compression-compaction-to-reduce-disk-usage/21258/1 "2014-12-15T14:20:14Z")
Hi list, We are dumping about 100M ~3KB documents a day into ElasticSearch 1.4.1 and indexing all fields (of which there are a few dozen). From a read perspective we perform dynamic queries which may return many results (all of which may be relevant, we don't use scoring) so we want to keep the \_source field. Obviously, this is taking a toll on our disk usage and we'd like to reduce that. Questions: 1. Is it possible for me to index JSON but set the \_source field myself? I would shove a protobuf or something similar on insert and on query I would revert it back to JSON 2. I understand that \_source is compressed, but I assume every document is compressed separately (our small documents don't benefit from that). Is there a way to somehow compress "across" documents to take advantage of the fact that our documents are extremely similar to one another? 3. Any other ideas? Thanks, Eran -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/f0588de1-b0e4-468f-ad25-b74f4abb444d%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/f0588de1-b0e4-468f-ad25-b74f4abb444d%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Robert\_Muir\_2](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/robert_muir_2/32/931_2.png) [@Robert\_Muir\_2](https://discuss.elastic.co/u/Robert_Muir_2)\ **Post date:** [December 15, 2014, 3:44pm UTC](https://discuss.elastic.co/t/custom--source-compression-compaction-to-reduce-disk-usage/21258/2 "2014-12-15T15:44:08Z")
On Mon, Dec 15, 2014 at 9:20 AM, Eran Duchan [pavius@gmail.com](mailto:pavius@gmail.com) wrote: > I understand that \_source is compressed, but I assume every document is > compressed separately (our small documents don't benefit from that). That is not the case, blocks of documents are compressed together: [https://lucene.apache.org/core/4\_10\_0/core/org/apache/lucene/codecs/lucene41/Lucene41StoredFieldsFormat.html](https://lucene.apache.org/core/4_10_0/core/org/apache/lucene/codecs/lucene41/Lucene41StoredFieldsFormat.html) -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAMUKNZXbii9TQWvfNbEx5QPMjhVfOVxSAS8S43eE-GOyzEfo%2BQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAMUKNZXbii9TQWvfNbEx5QPMjhVfOVxSAS8S43eE-GOyzEfo%2BQ%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Eran\_Duchan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/eran_duchan/32/1029_2.png) [@Eran\_Duchan](https://discuss.elastic.co/u/Eran_Duchan)\ **Post date:** [December 15, 2014, 4:49pm UTC](https://discuss.elastic.co/t/custom--source-compression-compaction-to-reduce-disk-usage/21258/3 "2014-12-15T16:49:35Z")
On Monday, December 15, 2014 5:44:38 PM UTC+2, Robert Muir wrote: > That is not the case, blocks of documents are compressed together: Thanks, Robert. I unscientifically swam around the code pivoting around this and saw that: 1. This isn't tweakable - I can't choose to compress in larger chunks 2. 2.0.0 will have an option to use deflate [https://github.com/elasticsearch/elasticsearch/pull/8863](https://github.com/elasticsearch/elasticsearch/pull/8863) for better compression So if I can't tweak \_source compression, can I shove a \_source of my own as posted originally in (1)? Eran -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/d036057a-7a04-4ead-a9c6-a91146be1708%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/d036057a-7a04-4ead-a9c6-a91146be1708%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Robert\_Muir\_2](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/robert_muir_2/32/931_2.png) [@Robert\_Muir\_2](https://discuss.elastic.co/u/Robert_Muir_2)\ **Post date:** [December 15, 2014, 5:01pm UTC](https://discuss.elastic.co/t/custom--source-compression-compaction-to-reduce-disk-usage/21258/4 "2014-12-15T17:01:22Z")
On Mon, Dec 15, 2014 at 11:49 AM, Eran Duchan [pavius@gmail.com](mailto:pavius@gmail.com) wrote: > On Monday, December 15, 2014 5:44:38 PM UTC+2, Robert Muir wrote: > > > That is not the case, blocks of documents are compressed together: > > Thanks, Robert. > > I unscientifically swam around the code pivoting around this and saw that: > > This isn't tweakable - I can't choose to compress in larger chunks > 2.0.0 will have an option to use deflate for better compression > > So if I can't tweak \_source compression, can I shove a \_source of my own as > posted originally in (1)? Its not really tweakable at all before lucene 5, thats why we added a higher compression option. Note this option is not just deflate but also uses a higher blocksize and other internal parameters. Using a higher blocksize (64kb) for deflate is really a simple workaround to get the feature out sooner than later, with the idea that people that choose BEST\_COMPRESSION are willing to sacrifice some retrieval speed. Increasing blocksize has a negative cost on retrieval performance and is not really the best way overall to get better compression when there is high redundancy across documents. In the future I hope we can add preset dictionary support for sharing across blocks. So the current blocksize should really be seen as an internal thing. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAMUKNZX8-SGzuNvvZ%3D\_-Sec\_%2Bq3svtLBE-\_d3L%2B70TR8Nm\_%3Drw%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAMUKNZX8-SGzuNvvZ%3D_-Sec_%2Bq3svtLBE-_d3L%2B70TR8Nm_%3Drw%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Eran\_Duchan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/eran_duchan/32/1029_2.png) [@Eran\_Duchan](https://discuss.elastic.co/u/Eran_Duchan)\ **Post date:** [December 15, 2014, 5:12pm UTC](https://discuss.elastic.co/t/custom--source-compression-compaction-to-reduce-disk-usage/21258/5 "2014-12-15T17:12:31Z")
Got it, thanks. Any insight on a custom \_source? Is this doable? -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/bcf7dae7-2164-43ae-b2bd-c44ed0d99aaa%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/bcf7dae7-2164-43ae-b2bd-c44ed0d99aaa%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Robert\_Muir\_2](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/robert_muir_2/32/931_2.png) [@Robert\_Muir\_2](https://discuss.elastic.co/u/Robert_Muir_2)\ **Post date:** [December 15, 2014, 5:38pm UTC](https://discuss.elastic.co/t/custom--source-compression-compaction-to-reduce-disk-usage/21258/6 "2014-12-15T17:38:53Z")
I dont understand what you hope to benefit from it. Using a binary encoding isn't going to improve the compression here really... I have done tests with it. On Mon, Dec 15, 2014 at 12:12 PM, Eran Duchan [pavius@gmail.com](mailto:pavius@gmail.com) wrote: > Got it, thanks. > Any insight on a custom \_source? Is this doable? > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/bcf7dae7-2164-43ae-b2bd-c44ed0d99aaa%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/bcf7dae7-2164-43ae-b2bd-c44ed0d99aaa%40googlegroups.com). > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAMUKNZUhtNHqG%2BW9KnKWfRzTEUkwG8JWo%2Bz6-%3DGUDkZ-eSu6-w%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAMUKNZUhtNHqG%2BW9KnKWfRzTEUkwG8JWo%2Bz6-%3DGUDkZ-eSu6-w%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Eran\_Duchan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/eran_duchan/32/1029_2.png) [@Eran\_Duchan](https://discuss.elastic.co/u/Eran_Duchan)\ **Post date:** [December 15, 2014, 5:53pm UTC](https://discuss.elastic.co/t/custom--source-compression-compaction-to-reduce-disk-usage/21258/7 "2014-12-15T17:53:23Z")
Seems to me that an uncompressed binary representation ala protobuf would be smaller than compressed JSON given our schema. If that were to prove correct, is it possibe to do this? I dont expect ElasticSearch to do anything except allowing me to control the contents of \_source. Eran -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/3593253a-ced8-40dd-aa1f-745edd71bef8%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/3593253a-ced8-40dd-aa1f-745edd71bef8%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Robert\_Muir\_2](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/robert_muir_2/32/931_2.png) [@Robert\_Muir\_2](https://discuss.elastic.co/u/Robert_Muir_2)\ **Post date:** [December 15, 2014, 5:59pm UTC](https://discuss.elastic.co/t/custom--source-compression-compaction-to-reduce-disk-usage/21258/8 "2014-12-15T17:59:10Z")
On Mon, Dec 15, 2014 at 12:53 PM, Eran Duchan [pavius@gmail.com](mailto:pavius@gmail.com) wrote: > Seems to me that an uncompressed binary representation ala protobuf would be smaller than compressed JSON given our schema. > > If that were to prove correct, is it possibe to do this? I dont expect Elasticsearch to do anything except allowing me to control the contents of \_source. The question is how much. for example in my tests it saves 2% with lz4 and 0.5% with deflate (the larger block size used there makes it super-not-worth-it). Its not worth the complexity IMO. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAMUKNZUMG4G6jF7Rw%3Dn9gMenQnm8nytnJ84WMW0K3CjTt%2Bc8TA%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAMUKNZUMG4G6jF7Rw%3Dn9gMenQnm8nytnJ84WMW0K3CjTt%2Bc8TA%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Eran\_Duchan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/eran_duchan/32/1029_2.png) [@Eran\_Duchan](https://discuss.elastic.co/u/Eran_Duchan)\ **Post date:** [December 15, 2014, 6:11pm UTC](https://discuss.elastic.co/t/custom--source-compression-compaction-to-reduce-disk-usage/21258/9 "2014-12-15T18:11:01Z")
Given those stats I totally agree, but this would have to vary given different schemas... That's why I'd like to at least experiment with it. Is this even possible through the public http interface? -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/29797a61-473e-48f0-9d00-acd82cbe26bc%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/29797a61-473e-48f0-9d00-acd82cbe26bc%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![rmuir](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rmuir/32/44949_2.png) [@rmuir](https://discuss.elastic.co/u/rmuir)\ **Post date:** [December 15, 2014, 6:27pm UTC](https://discuss.elastic.co/t/custom--source-compression-compaction-to-reduce-disk-usage/21258/10 "2014-12-15T18:27:36Z")
If you want to do such experiments, it will be hard to do with ES, since you would have to plumb a ton of code to even get the results. Instead I write lucene code to test these things out. This also makes the benchmark fast since i dont "index" anything so there is no real flushing or merging going on to make benchmarking more difficult (when i want to measure performance of things like merge, i force it to happen at predictable intervals to keep index size comparisons valid). On Mon, Dec 15, 2014 at 1:11 PM, Eran Duchan [pavius@gmail.com](mailto:pavius@gmail.com) wrote: > Given those stats I totally agree, but this would have to vary given different schemas... That's why I'd like to at least experiment with it. Is this even possible through the public http interface? > > -- > You received this message because you are subscribed to the Google Groups "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/29797a61-473e-48f0-9d00-acd82cbe26bc%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/29797a61-473e-48f0-9d00-acd82cbe26bc%40googlegroups.com). > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAOdYfZW0K13wCqUwd3jLF0ifRkQsW\_v9cnR5D75zdGTk-%2BWP2w%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAOdYfZW0K13wCqUwd3jLF0ifRkQsW_v9cnR5D75zdGTk-%2BWP2w%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Eran\_Duchan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/eran_duchan/32/1029_2.png) [@Eran\_Duchan](https://discuss.elastic.co/u/Eran_Duchan)\ **Post date:** [December 15, 2014, 8:56pm UTC](https://discuss.elastic.co/t/custom--source-compression-compaction-to-reduce-disk-usage/21258/11 "2014-12-15T20:56:21Z")
Thanks for the pointers. I just realized I can disable \_source and "store" a field with the encoded data (D'oh). If I find anything semi-intelligent during my tests, I'll report back. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/bd777d03-06d9-416b-8366-1b6b1f6e1302%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/bd777d03-06d9-416b-8366-1b6b1f6e1302%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Itamar\_Syn\_Hershko](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/itamar_syn_hershko/32/725_2.png) [@Itamar\_Syn\_Hershko](https://discuss.elastic.co/u/Itamar_Syn_Hershko)\ **Post date:** [December 15, 2014, 9:14pm UTC](https://discuss.elastic.co/t/custom--source-compression-compaction-to-reduce-disk-usage/21258/12 "2014-12-15T21:14:01Z")
I'm pretty sure you'll lose cross-document compression that way, which is highly noticable on lots of 3k large documents -- Itamar Syn-Hershko [http://code972.com](http://code972.com) | @synhershko [https://twitter.com/synhershko](https://twitter.com/synhershko) Freelance Developer & Consultant Author of RavenDB in Action [http://manning.com/synhershko/](http://manning.com/synhershko/) On Mon, Dec 15, 2014 at 10:56 PM, Eran Duchan [pavius@gmail.com](mailto:pavius@gmail.com) wrote: > Thanks for the pointers. I just realized I can disable \_source and "store" > a field with the encoded data (D'oh). If I find anything semi-intelligent > during my tests, I'll report back. > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/bd777d03-06d9-416b-8366-1b6b1f6e1302%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/bd777d03-06d9-416b-8366-1b6b1f6e1302%40googlegroups.com) > [https://groups.google.com/d/msgid/elasticsearch/bd777d03-06d9-416b-8366-1b6b1f6e1302%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/bd777d03-06d9-416b-8366-1b6b1f6e1302%40googlegroups.com?utm_medium=email&utm_source=footer) > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZttApz%3D6kG%2B\_Pw19V%3Db-%2Bbvh-deAsZwRZ3TeeqXuzGq5g%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZttApz%3D6kG%2B_Pw19V%3Db-%2Bbvh-deAsZwRZ3TeeqXuzGq5g%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Eran\_Duchan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/eran_duchan/32/1029_2.png) [@Eran\_Duchan](https://discuss.elastic.co/u/Eran_Duchan)\ **Post date:** [December 18, 2014, 9:26am UTC](https://discuss.elastic.co/t/custom--source-compression-compaction-to-reduce-disk-usage/21258/13 "2014-12-18T09:26:35Z")
After a bit of testing, it was found that \_source takes up 37% of the total disk space. Given everything said here (cross document compression, future support for better compression), we decided to leave this as is. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/69b606dd-ab53-402c-b409-97128840ab59%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/69b606dd-ab53-402c-b409-97128840ab59%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 12:42am UTC](https://discuss.elastic.co/t/custom--source-compression-compaction-to-reduce-disk-usage/21258/14 "2017-07-06T00:42:57Z")