# Aggressive index compression
**URL:** https://discuss.elastic.co/t/aggressive-index-compression/9049
**Category:** Elasticsearch
**Created:** [September 18, 2012, 1:09pm UTC](https://discuss.elastic.co/t/aggressive-index-compression/9049 "2012-09-18T13:09:25Z")
**Posts on this page:** 9
**Page:** 1
### Author:  [@Robin\_Verlangen](https://discuss.elastic.co/u/Robin_Verlangen)
#### Post date: [September 18, 2012, 1:09pm UTC](https://discuss.elastic.co/t/aggressive-index-compression/9049/1 "2012-09-18T13:09:25Z")
Hi there,
We're willing to index lots of data, however after a certain period of time
it doesn't have to be as "hot" as data from the past week. What do you
experts think of the following:
1. Open index
2. Write lots of data into it
> > Index becomes less important after 7 days
1. Close index
2. (g)zip the index
> > Index remains gzip
1. Request for search on index
2. Un(g)zip the index
3. Open the index
4. Perform the search
Is there anything I miss here, that would cause problems? A quick test on a
single node worked perfectly. The default compression doesn't help us
enough.
Best regards,
Robin Verlangen
_Software engineer_
\*
\*
W [http://www.robinverlangen.nl](http://www.robinverlangen.nl)
E robin@us2.nl
Disclaimer: The information contained in this message and attachments is
intended solely for the attention and use of the named addressee and may be
confidential. If you are not the intended recipient, you are reminded that
the information remains the property of the sender. You must not use,
disclose, distribute, copy, print or rely on this e-mail. If you have
received this message in error, please contact the sender immediately and
irrevocably delete this message and any copies.
--
---
### Author:  [@kimchy](https://discuss.elastic.co/u/kimchy)
#### Post date: [September 18, 2012, 7:47pm UTC](https://discuss.elastic.co/t/aggressive-index-compression/9049/2 "2012-09-18T19:47:46Z")
The relatively new compression option should do the trick without the need to gzip your index (you won't gain that much). More details here: [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/blog/2012/06/25/0.19.5-released.html).
On Sep 18, 2012, at 3:09 PM, Robin Verlangen [robin@us2.nl](mailto:robin@us2.nl) wrote:
> Hi there,
>
> We're willing to index lots of data, however after a certain period of time it doesn't have to be as "hot" as data from the past week. What do you experts think of the following:
>
> 1. Open index
> 2. Write lots of data into it
>
> > > Index becomes less important after 7 days
>
> 1. Close index
> 2. (g)zip the index
>
> > > Index remains gzip
>
> 1. Request for search on index
> 2. Un(g)zip the index
> 3. Open the index
> 4. Perform the search
>
> Is there anything I miss here, that would cause problems? A quick test on a single node worked perfectly. The default compression doesn't help us enough.
>
> Best regards,
>
> Robin Verlangen
> Software engineer
>
> W [http://www.robinverlangen.nl](http://www.robinverlangen.nl)
> E robin@us2.nl
>
> Disclaimer: The information contained in this message and attachments is intended solely for the attention and use of the named addressee and may be confidential. If you are not the intended recipient, you are reminded that the information remains the property of the sender. You must not use, disclose, distribute, copy, print or rely on this e-mail. If you have received this message in error, please contact the sender immediately and irrevocably delete this message and any copies.
>
> --
--
---
### Author:  [@Robin\_Verlangen](https://discuss.elastic.co/u/Robin_Verlangen)
#### Post date: [September 19, 2012, 7:29am UTC](https://discuss.elastic.co/t/aggressive-index-compression/9049/3 "2012-09-19T07:29:56Z")
Thank you for the reference, however I was already aware of those options.
A quick benchmark gave us indication we could still win a lot: I'll publish
the details in here soon!
Best regards,
Robin Verlangen
_Software engineer_
\*
\*
W [http://www.robinverlangen.nl](http://www.robinverlangen.nl)
E robin@us2.nl
Disclaimer: The information contained in this message and attachments is
intended solely for the attention and use of the named addressee and may be
confidential. If you are not the intended recipient, you are reminded that
the information remains the property of the sender. You must not use,
disclose, distribute, copy, print or rely on this e-mail. If you have
received this message in error, please contact the sender immediately and
irrevocably delete this message and any copies.
2012/9/18 Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com)
> The relatively new compression option should do the trick without the need
> to gzip your index (you won't gain that much). More details here:
> [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/blog/2012/06/25/0.19.5-released.html).
>
> On Sep 18, 2012, at 3:09 PM, Robin Verlangen [robin@us2.nl](mailto:robin@us2.nl) wrote:
>
> Hi there,
>
> We're willing to index lots of data, however after a certain period of
> time it doesn't have to be as "hot" as data from the past week. What do you
> experts think of the following:
>
> 1. Open index
> 2. Write lots of data into it
>
> > > Index becomes less important after 7 days
>
> 1. Close index
> 2. (g)zip the index
>
> > > Index remains gzip
>
> 1. Request for search on index
> 2. Un(g)zip the index
> 3. Open the index
> 4. Perform the search
>
> Is there anything I miss here, that would cause problems? A quick test on
> a single node worked perfectly. The default compression doesn't help us
> enough.
>
> Best regards,
>
> Robin Verlangen
> _Software engineer_
> \*
> \*
> W [http://www.robinverlangen.nl](http://www.robinverlangen.nl)
> E robin@us2.nl
>
> Disclaimer: The information contained in this message and attachments is
> intended solely for the attention and use of the named addressee and may be
> confidential. If you are not the intended recipient, you are reminded that
> the information remains the property of the sender. You must not use,
> disclose, distribute, copy, print or rely on this e-mail. If you have
> received this message in error, please contact the sender immediately and
> irrevocably delete this message and any copies.
>
> --
>
> --
--
---
### Author:  [@Robin\_Verlangen](https://discuss.elastic.co/u/Robin_Verlangen)
#### Post date: [September 19, 2012, 8:13am UTC](https://discuss.elastic.co/t/aggressive-index-compression/9049/4 "2012-09-19T08:13:55Z")
I published my findings on Elasticsearch compression on my personal blog
with some in-depth information about the benchmark:
[http://www.robinverlangen.nl/index/view/50597e5876ad1-6e7e5f/elasticsearch-compression-benchmark.html](http://www.robinverlangen.nl/index/view/50597e5876ad1-6e7e5f/elasticsearch-compression-benchmark.html)
Best regards,
Robin Verlangen
_Software engineer_
\*
\*
W [http://www.robinverlangen.nl](http://www.robinverlangen.nl)
E robin@us2.nl
Disclaimer: The information contained in this message and attachments is
intended solely for the attention and use of the named addressee and may be
confidential. If you are not the intended recipient, you are reminded that
the information remains the property of the sender. You must not use,
disclose, distribute, copy, print or rely on this e-mail. If you have
received this message in error, please contact the sender immediately and
irrevocably delete this message and any copies.
2012/9/19 Robin Verlangen [robin@us2.nl](mailto:robin@us2.nl)
> Thank you for the reference, however I was already aware of those options.
> A quick benchmark gave us indication we could still win a lot: I'll publish
> the details in here soon!
>
> Best regards,
>
> Robin Verlangen
> _Software engineer_
> \*
> \*
> W [http://www.robinverlangen.nl](http://www.robinverlangen.nl)
> E robin@us2.nl
>
> Disclaimer: The information contained in this message and attachments is
> intended solely for the attention and use of the named addressee and may be
> confidential. If you are not the intended recipient, you are reminded that
> the information remains the property of the sender. You must not use,
> disclose, distribute, copy, print or rely on this e-mail. If you have
> received this message in error, please contact the sender immediately and
> irrevocably delete this message and any copies.
>
> 2012/9/18 Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com)
>
> > The relatively new compression option should do the trick without the
> > need to gzip your index (you won't gain that much). More details here:
> > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/blog/2012/06/25/0.19.5-released.html).
> >
> > On Sep 18, 2012, at 3:09 PM, Robin Verlangen [robin@us2.nl](mailto:robin@us2.nl) wrote:
> >
> > Hi there,
> >
> > We're willing to index lots of data, however after a certain period of
> > time it doesn't have to be as "hot" as data from the past week. What do you
> > experts think of the following:
> >
> > 1. Open index
> > 2. Write lots of data into it
> >
> > > > Index becomes less important after 7 days
> >
> > 1. Close index
> > 2. (g)zip the index
> >
> > > > Index remains gzip
> >
> > 1. Request for search on index
> > 2. Un(g)zip the index
> > 3. Open the index
> > 4. Perform the search
> >
> > Is there anything I miss here, that would cause problems? A quick test on
> > a single node worked perfectly. The default compression doesn't help us
> > enough.
> >
> > Best regards,
> >
> > Robin Verlangen
> > _Software engineer_
> > \*
> > \*
> > W [http://www.robinverlangen.nl](http://www.robinverlangen.nl)
> > E robin@us2.nl
> >
> > Disclaimer: The information contained in this message and attachments is
> > intended solely for the attention and use of the named addressee and may be
> > confidential. If you are not the intended recipient, you are reminded that
> > the information remains the property of the sender. You must not use,
> > disclose, distribute, copy, print or rely on this e-mail. If you have
> > received this message in error, please contact the sender immediately and
> > irrevocably delete this message and any copies.
> >
> > --
> >
> > --
--
---
### Author:  [@Clinton\_Gormley](https://discuss.elastic.co/u/Clinton_Gormley)
#### Post date: [September 19, 2012, 8:26am UTC](https://discuss.elastic.co/t/aggressive-index-compression/9049/5 "2012-09-19T08:26:20Z")
Hi Robin
On Wed, 2012-09-19 at 10:13 +0200, Robin Verlangen wrote:
> I published my findings on Elasticsearch compression on my personal
> blog with some in-depth information about the benchmark:
> [http://www.robinverlangen.nl/index/view/50597e5876ad1-6e7e5f/elasticsearch-compression-benchmark.html](http://www.robinverlangen.nl/index/view/50597e5876ad1-6e7e5f/elasticsearch-compression-benchmark.html)
Are your numbers correct? You say that an index without compressions was
3,282 MB but an index with compression was 15,400 MB ??? ie 5 times
BIGGER?
clint
>
--
---
### Author:  [@Robin\_Verlangen](https://discuss.elastic.co/u/Robin_Verlangen)
#### Post date: [September 19, 2012, 8:36am UTC](https://discuss.elastic.co/t/aggressive-index-compression/9049/6 "2012-09-19T08:36:45Z")
That was a stupid typo, of course 1540MB, I updated the PDF file attached:
[http://www.robinverlangen.nl/assets/blog/ES-benchmark---Compress-indexes.pdf](http://www.robinverlangen.nl/assets/blog/ES-benchmark---Compress-indexes.pdf)
Best regards,
Robin Verlangen
_Software engineer_
\*
\*
W [http://www.robinverlangen.nl](http://www.robinverlangen.nl)
E robin@us2.nl
Disclaimer: The information contained in this message and attachments is
intended solely for the attention and use of the named addressee and may be
confidential. If you are not the intended recipient, you are reminded that
the information remains the property of the sender. You must not use,
disclose, distribute, copy, print or rely on this e-mail. If you have
received this message in error, please contact the sender immediately and
irrevocably delete this message and any copies.
2012/9/19 Clinton Gormley [clint@traveljury.com](mailto:clint@traveljury.com)
> Hi Robin
>
> On Wed, 2012-09-19 at 10:13 +0200, Robin Verlangen wrote:
>
> > I published my findings on Elasticsearch compression on my personal
> > blog with some in-depth information about the benchmark:
>
> [http://www.robinverlangen.nl/index/view/50597e5876ad1-6e7e5f/elasticsearch-compression-benchmark.html](http://www.robinverlangen.nl/index/view/50597e5876ad1-6e7e5f/elasticsearch-compression-benchmark.html)
>
> Are your numbers correct? You say that an index without compressions was
> 3,282 MB but an index with compression was 15,400 MB ??? ie 5 times
> BIGGER?
>
> clint
>
> >
>
> --
--
---
### Author:  [@Radim](https://discuss.elastic.co/u/Radim)
#### Post date: [September 19, 2012, 8:02pm UTC](https://discuss.elastic.co/t/aggressive-index-compression/9049/7 "2012-09-19T20:02:40Z")
Thanks Robin -- seeing some real numbers is always refreshing! IMO
Elasticsearch documentation dearly lacks some "ballpark figures" of
what to expect under common scenarios. Together with some general info
on how things are expected to scale (constant/linear/sublinear...)
that would already help newcomers a lot.
Btw any chance you might also be comparing query times in your setup?
Would the no/compression/tv affect that at all? (ignoring the offline
ZIP option, of course)
Best,
Radim
On Sep 19, 10:36 am, Robin Verlangen [ro...@us2.nl](mailto:ro...@us2.nl) wrote:
> That was a stupid typo, of course 1540MB, I updated the PDF file attached:[http://www.robinverlangen.nl/assets/blog/ES-benchmark---Compress-inde](http://www.robinverlangen.nl/assets/blog/ES-benchmark---Compress-inde)...
>
> Best regards,
>
> Robin Verlangen
> _Software engineer_
> \*
> \*
> Whttp://www.robinverlangen.nl
> E ro...@us2.nl
>
> Disclaimer: The information contained in this message and attachments is
> intended solely for the attention and use of the named addressee and may be
> confidential. If you are not the intended recipient, you are reminded that
> the information remains the property of the sender. You must not use,
> disclose, distribute, copy, print or rely on this e-mail. If you have
> received this message in error, please contact the sender immediately and
> irrevocably delete this message and any copies.
>
> 2012/9/19 Clinton Gormley [cl...@traveljury.com](mailto:cl...@traveljury.com)
>
> > Hi Robin
>
> > On Wed, 2012-09-19 at 10:13 +0200, Robin Verlangen wrote:
> >
> > > I published my findings on Elasticsearch compression on my personal
> > > blog with some in-depth information about the benchmark:
>
> > [http://www.robinverlangen.nl/index/view/50597e5876ad1-6e7e5f/elastics](http://www.robinverlangen.nl/index/view/50597e5876ad1-6e7e5f/elastics)...
>
> > Are your numbers correct? You say that an index without compressions was
> > 3,282 MB but an index with compression was 15,400 MB ??? ie 5 times
> > BIGGER?
>
> > clint
>
> > --
--
---
### Author:  [@Robin\_Verlangen](https://discuss.elastic.co/u/Robin_Verlangen)
#### Post date: [September 20, 2012, 6:39am UTC](https://discuss.elastic.co/t/aggressive-index-compression/9049/8 "2012-09-20T06:39:04Z")
Hi Radim,
We'll get into that later. Our application (CloudPelican) is going to
gather lots and lots of data from all kinds of different sources. We
already picked Elasticsearch out of Solor, Solandra, raw Cassandra and
Lucene for our indexing process. First important thing for us was to
determine how much storage overhead was involved. Query times are relevant,
but probably tuneable with lots of parameters.
Once we have more I'll update you over here, or you can just stay in touch
with my blog.
Best regards,
Robin Verlangen
_Software engineer_
\*
\*
W [http://www.robinverlangen.nl](http://www.robinverlangen.nl)
E robin@us2.nl
Disclaimer: The information contained in this message and attachments is
intended solely for the attention and use of the named addressee and may be
confidential. If you are not the intended recipient, you are reminded that
the information remains the property of the sender. You must not use,
disclose, distribute, copy, print or rely on this e-mail. If you have
received this message in error, please contact the sender immediately and
irrevocably delete this message and any copies.
2012/9/19 Radim [me@radimrehurek.com](mailto:me@radimrehurek.com)
> Thanks Robin -- seeing some real numbers is always refreshing! IMO
> Elasticsearch documentation dearly lacks some "ballpark figures" of
> what to expect under common scenarios. Together with some general info
> on how things are expected to scale (constant/linear/sublinear...)
> that would already help newcomers a lot.
>
> Btw any chance you might also be comparing query times in your setup?
> Would the no/compression/tv affect that at all? (ignoring the offline
> ZIP option, of course)
>
> Best,
> Radim
>
> On Sep 19, 10:36 am, Robin Verlangen [ro...@us2.nl](mailto:ro...@us2.nl) wrote:
>
> > That was a stupid typo, of course 1540MB, I updated the PDF file
> > attached:
> > [http://www.robinverlangen.nl/assets/blog/ES-benchmark---Compress-inde](http://www.robinverlangen.nl/assets/blog/ES-benchmark---Compress-inde)...
> >
> > Best regards,
> >
> > Robin Verlangen
> > _Software engineer_
> > \*
> > \*
> > Whttp://www.robinverlangen.nl
> > E ro...@us2.nl
> >
> > Disclaimer: The information contained in this message and attachments is
> > intended solely for the attention and use of the named addressee and may
> > be
> > confidential. If you are not the intended recipient, you are reminded
> > that
> > the information remains the property of the sender. You must not use,
> > disclose, distribute, copy, print or rely on this e-mail. If you have
> > received this message in error, please contact the sender immediately and
> > irrevocably delete this message and any copies.
> >
> > 2012/9/19 Clinton Gormley [cl...@traveljury.com](mailto:cl...@traveljury.com)
> >
> > > Hi Robin
> >
> > > On Wed, 2012-09-19 at 10:13 +0200, Robin Verlangen wrote:
> > >
> > > > I published my findings on Elasticsearch compression on my personal
> > > > blog with some in-depth information about the benchmark:
> >
> > > [http://www.robinverlangen.nl/index/view/50597e5876ad1-6e7e5f/elastics](http://www.robinverlangen.nl/index/view/50597e5876ad1-6e7e5f/elastics).
> > > ..
> >
> > > Are your numbers correct? You say that an index without compressions
> > > was
> > > 3,282 MB but an index with compression was 15,400 MB ??? ie 5 times
> > > BIGGER?
> >
> > > clint
> >
> > > --
>
> --
--
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 6, 2017, 3:12am UTC](https://discuss.elastic.co/t/aggressive-index-compression/9049/9 "2017-07-06T03:12:11Z")