# Compression in Elasticsearch documents **URL:** \ **Category:** Elasticsearch\ **Created:** [April 14, 2015, 4:47pm UTC](https://discuss.elastic.co/t/compression-in-elasticsearch-documents/23235 "2015-04-14T16:47:59Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![ajay\_bh111](https://avatars.discourse-cdn.com/v4/letter/a/f08c70/32.png) [@ajay\_bh111](https://discuss.elastic.co/u/ajay_bh111)\ **Post date:** [April 14, 2015, 4:47pm UTC](https://discuss.elastic.co/t/compression-in-elasticsearch-documents/23235/1 "2015-04-14T16:47:59Z")
I would like to know if Elasticsearch documents/indices are stored in compressed format on disk . If yes, what type of compression options are available and it's performance overheads. and if these compression options are configurable. Thanks Ajay -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/8c63c25f-4f49-47f4-8d0a-772d3301f45c%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/8c63c25f-4f49-47f4-8d0a-772d3301f45c%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![jpountz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jpountz/32/45836_2.png) [@jpountz](https://discuss.elastic.co/u/jpountz)\ **Post date:** [April 14, 2015, 5:13pm UTC](https://discuss.elastic.co/t/compression-in-elasticsearch-documents/23235/2 "2015-04-14T17:13:18Z")
Hi, Data are both duplicated to suit different access patterns and compressed. There are so many compression algorithms in-place that it would be hard to be exhaustive, but we have for instance Frame-Of-Reference compression for postings lists, LZ4 for the document store, bit packing for numeric doc values, ... There are no configurations options available to configure compression besides disabling features that you don't need (such as norms on fields that you don't score on). In the next major version of elasticsearch (2.0) there will be a setting to enable heavier compression though (which in practice will use DEFLATE instead of LZ4 for the document store): > > > upgrades lucene to latest, and supports the BEST\_COMPRESSION parameter now suppo…rted (with backwards compatibility, etc) in Lucene. This option uses deflate, tuned for highly compressible data. > > \`index.codec\`:: > The \`default\` value compresses stored data with LZ4 compression, but > this can be set to \`best\_compression\` for a higher compression ratio, > at the expense of slower stored fields performance. > > IMO its safest to implement as a named codec here, because ES already has logic to handle this correctly, and because its unrealistic to have a plethora of options to Lucene's default codec... we are practically limited in Lucene to what we can support with back compat, so I don't think we should overengineer this and add additional unnecessary plumbing. > > See also: > https://issues.apache.org/jira/browse/LUCENE-5914 > https://issues.apache.org/jira/browse/LUCENE-6089 > https://issues.apache.org/jira/browse/LUCENE-6090 > https://issues.apache.org/jira/browse/LUCENE-6100 On Tue, Apr 14, 2015 at 6:47 PM, [ajay.bh111@gmail.com](mailto:ajay.bh111@gmail.com) wrote: > I would like to know if Elasticsearch documents/indices are stored in > compressed format on disk . If yes, what type of compression options are > available and it's performance overheads. > > and if these compression options are configurable. > > Thanks > Ajay > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/8c63c25f-4f49-47f4-8d0a-772d3301f45c%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/8c63c25f-4f49-47f4-8d0a-772d3301f45c%40googlegroups.com) > [https://groups.google.com/d/msgid/elasticsearch/8c63c25f-4f49-47f4-8d0a-772d3301f45c%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/8c63c25f-4f49-47f4-8d0a-772d3301f45c%40googlegroups.com?utm_medium=email&utm_source=footer) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- Adrien -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAO5%3DkAiF5uRZmGCocKgjeiuBahpsc1iMZ-7XkQWFzWK3hVWPvg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAO5%3DkAiF5uRZmGCocKgjeiuBahpsc1iMZ-7XkQWFzWK3hVWPvg%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![ajay\_bh111](https://avatars.discourse-cdn.com/v4/letter/a/f08c70/32.png) [@ajay\_bh111](https://discuss.elastic.co/u/ajay_bh111)\ **Post date:** [April 14, 2015, 5:35pm UTC](https://discuss.elastic.co/t/compression-in-elasticsearch-documents/23235/3 "2015-04-14T17:35:34Z")
Hi Adrian Thanks for quick response. When I loaded nearly 45m documents of test data with 3 replicas (each document approx 2K+ bytes in size), I got following info on storage: _health status index pri rep docs.count docs.deleted store.size pri.store.size_ _green open test\_insert 5 3 44985382 0 414.9gb 106.4gb_ This indicates there was hardly any compression on physical storage\*.\* Hence my question. How do I find /estimate how much storage would be used for X number of documents of average size of Y kilobytes each. From above result, it appears to be no compression at all on all stored data. Thanks Ajay On Tuesday, April 14, 2015 at 1:13:37 PM UTC-4, Adrien Grand wrote: > Hi, > > Data are both duplicated to suit different access patterns and compressed. > There are so many compression algorithms in-place that it would be hard to > be exhaustive, but we have for instance Frame-Of-Reference compression for > postings lists, LZ4 for the document store, bit packing for numeric doc > values, ... > > There are no configurations options available to configure compression > besides disabling features that you don't need (such as norms on fields > that you don't score on). In the next major version of elasticsearch (2.0) > there will be a setting to enable heavier compression though (which in > practice will use DEFLATE instead of LZ4 for the document store): > [Add `best_compression` option for indices by rmuir · Pull Request #8863 · elastic/elasticsearch · GitHub](https://github.com/elastic/elasticsearch/pull/8863) > > On Tue, Apr 14, 2015 at 6:47 PM, \<[ajay....@gmail.com](mailto:ajay....@gmail.com) \\> wrote: > > > I would like to know if Elasticsearch documents/indices are stored in > > compressed format on disk . If yes, what type of compression options are > > available and it's performance overheads. > > > > and if these compression options are configurable. > > > > Thanks > > Ajay > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/8c63c25f-4f49-47f4-8d0a-772d3301f45c%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/8c63c25f-4f49-47f4-8d0a-772d3301f45c%40googlegroups.com) > > [https://groups.google.com/d/msgid/elasticsearch/8c63c25f-4f49-47f4-8d0a-772d3301f45c%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/8c63c25f-4f49-47f4-8d0a-772d3301f45c%40googlegroups.com?utm_medium=email&utm_source=footer) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > -- > Adrien -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/50726923-3199-457b-a53e-24978cb94510%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/50726923-3199-457b-a53e-24978cb94510%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![jpountz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jpountz/32/45836_2.png) [@jpountz](https://discuss.elastic.co/u/jpountz)\ **Post date:** [April 15, 2015, 7:14am UTC](https://discuss.elastic.co/t/compression-in-elasticsearch-documents/23235/4 "2015-04-15T07:14:22Z")
On Tue, Apr 14, 2015 at 7:35 PM, [ajay.bh111@gmail.com](mailto:ajay.bh111@gmail.com) wrote: > Hi Adrian > > Thanks for quick response. > > When I loaded nearly 45m documents of test data with 3 replicas (each > document approx 2K+ bytes in size), I got following info on storage: > > _health status index pri rep docs.count docs.deleted store.size > pri.store.size_ > > _green open test\_insert 5 3 44985382 0 > 414.9gb 106.4gb_ > > This indicates there was hardly any compression on physical storage\*.\* > Hence my question. How do I find /estimate how much storage would be used > for X number of documents of average size of Y kilobytes each. From above > result, it appears to be no compression at all on all stored data. Compression ratios depend so much on the data that you can't really know what the compression ratio will be without indexing sample documents. However, once you indexed enough documents (eg. 100k), you can expect the store size to keep growing linearly with the number of documents. Most of time the largest part of the index is the document store. In your case I assume that LZ4 is too lightweight a compression algorithm to manage to compress your data efficiently. The high compression option which is coming in elasticsearch 2.0 might help. -- Adrien -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAO5%3DkAhehRg9QDNCK-TO%2BKYnX3T%2B5BH9QEM5nUi21u%2BgqBQEFg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAO5%3DkAhehRg9QDNCK-TO%2BKYnX3T%2B5BH9QEM5nUi21u%2BgqBQEFg%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![cdahlqvist](https://avatars.discourse-cdn.com/v4/letter/c/9fc348/32.png) [@cdahlqvist](https://discuss.elastic.co/u/cdahlqvist)\ **Post date:** [April 15, 2015, 8:25am UTC](https://discuss.elastic.co/t/compression-in-elasticsearch-documents/23235/5 "2015-04-15T08:25:22Z")
Hi, How much space the data takes up on disk in Elasticsearch depends a lot on your mappings. In addition to storing the source in the \_source field, all fields are by default also copied over to the \_all field to allow free text search across all fields. In addition to this Elasticsearch also indexes all the fields in the source document, sometimes in multiple ways, which also takes up space. The amount of data Elasticsearch need to store can therefore grow quite a bit before compression is applied. You might be able to reduce the indexed size on disk by ensuring your mappings are as efficient as possible, e.g. by disabling the \_all field if you do not need it. Best regards, Christian On Tuesday, April 14, 2015 at 5:47:59 PM UTC+1, [ajay....@gmail.com](mailto:ajay....@gmail.com) wrote: > I would like to know if Elasticsearch documents/indices are stored in > compressed format on disk . If yes, what type of compression options are > available and it's performance overheads. > > and if these compression options are configurable. > > Thanks > Ajay -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/fbeafe91-6c20-4e4c-9eff-94d7aa40e381%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/fbeafe91-6c20-4e4c-9eff-94d7aa40e381%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 12:19am UTC](https://discuss.elastic.co/t/compression-in-elasticsearch-documents/23235/6 "2017-07-06T00:19:41Z")