# Can I use gzip or some other compression algorithms when I create snapshots to HDFS? **URL:** \ **Category:** Elasticsearch\ **Created:** [July 3, 2019, 3:43am UTC](https://discuss.elastic.co/t/can-i-use-gzip-or-some-other-compression-algorithms-when-i-create-snapshots-to-hdfs/188600 "2019-07-03T03:43:23Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![wangxr1985](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/wangxr1985/32/117798_2.png) [@wangxr1985](https://discuss.elastic.co/u/wangxr1985)\ **Post date:** [July 3, 2019, 3:43am UTC](https://discuss.elastic.co/t/can-i-use-gzip-or-some-other-compression-algorithms-when-i-create-snapshots-to-hdfs/188600/1 "2019-07-03T03:43:23Z")
This is the document of repository-hdfs: [https://www.elastic.co/guide/en/elasticsearch/plugins/master/repository-hdfs-config.html](https://www.elastic.co/guide/en/elasticsearch/plugins/master/repository-hdfs-config.html) I can't find any compression algorithms arguments about HDFS when I use snapshot to do es data backup. I hope to use hdfs gzip to reduce the size of my backup data,because the "compress" argument of es snapshot seems to be unhelpful. Both with it and without it, the snapshot size eqauls to the raw es index size. ---
**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\ **Post date:** [July 3, 2019, 8:36am UTC](https://discuss.elastic.co/t/can-i-use-gzip-or-some-other-compression-algorithms-when-i-create-snapshots-to-hdfs/188600/2 "2019-07-03T08:36:59Z")
gzip won't help you too much, as the data within the segments already is compressed. The compress flag is also enabled by default and only refers to the metadata. Have you read the [Tune for disk usage](https://www.elastic.co/guide/en/elasticsearch/reference/7.2/tune-for-disk-usage.html) documentation, that talks about tricks to reduce your overall dataset? --Alex ---
**Author:** ![wangxr1985](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/wangxr1985/32/117798_2.png) [@wangxr1985](https://discuss.elastic.co/u/wangxr1985)\ **Post date:** [July 15, 2019, 11:03am UTC](https://discuss.elastic.co/t/can-i-use-gzip-or-some-other-compression-algorithms-when-i-create-snapshots-to-hdfs/188600/3 "2019-07-15T11:03:00Z")
I try to dump data from ES to text file and compare the size between them, and find that text file size equals to the ES index size. And then I try to use logstash to import data from ES to hdfs (with compression =\> "gzip" setting),the gzip file size in hdfs is 12% of ES index size. And then I try to reindex the data from one ES index with default codec setting to another ES index with "best\_compression" codec setting. the size of "best\_compression" index is 88% of the default setting index. ---
**Author:** ![wangxr1985](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/wangxr1985/32/117798_2.png) [@wangxr1985](https://discuss.elastic.co/u/wangxr1985)\ **Post date:** [July 15, 2019, 11:15am UTC](https://discuss.elastic.co/t/can-i-use-gzip-or-some-other-compression-algorithms-when-i-create-snapshots-to-hdfs/188600/4 "2019-07-15T11:15:50Z")
So I think data in ES index is organized almost as same as data in a text file, and segment merging 、data/metadata compression is unhelpful. Just gzip or other compression method can really reduce the size. ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [July 15, 2019, 1:29pm UTC](https://discuss.elastic.co/t/can-i-use-gzip-or-some-other-compression-algorithms-when-i-create-snapshots-to-hdfs/188600/5 "2019-07-15T13:29:02Z")
A backup (snapshot) and a json dump are definitely not the same thing. You can't compare both things. With a json dump, you can restore the dump and it will be ready to use where reading a json dump requires to reindex everything. Not the same data structures. ---
**Author:** ![wangxr1985](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/wangxr1985/32/117798_2.png) [@wangxr1985](https://discuss.elastic.co/u/wangxr1985)\ **Post date:** [July 16, 2019, 2:30pm UTC](https://discuss.elastic.co/t/can-i-use-gzip-or-some-other-compression-algorithms-when-i-create-snapshots-to-hdfs/188600/6 "2019-07-16T14:30:11Z")
I prefer to use snapshot ,the only issue is the backup data size. I don't want to compare it with json dump, but when I use json dump I can find a better way to reduce the size . I don't understand why snapshot does't support compession like gzip, I think the size is very important when we do data backup. ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [July 16, 2019, 2:44pm UTC](https://discuss.elastic.co/t/can-i-use-gzip-or-some-other-compression-algorithms-when-i-create-snapshots-to-hdfs/188600/7 "2019-07-16T14:44:23Z")
Did you try to gzip a snapshot? ---
**Author:** ![wangxr1985](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/wangxr1985/32/117798_2.png) [@wangxr1985](https://discuss.elastic.co/u/wangxr1985)\ **Post date:** [July 19, 2019, 11:29am UTC](https://discuss.elastic.co/t/can-i-use-gzip-or-some-other-compression-algorithms-when-i-create-snapshots-to-hdfs/188600/8 "2019-07-19T11:29:18Z")
I don't know how to do that when I use repository-hdfs to do snapshot to hdfs. ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [July 19, 2019, 11:53am UTC](https://discuss.elastic.co/t/can-i-use-gzip-or-some-other-compression-algorithms-when-i-create-snapshots-to-hdfs/188600/9 "2019-07-19T11:53:17Z")
snapshot to shared FS instead then gzip the directory. ---
**Author:** ![wangxr1985](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/wangxr1985/32/117798_2.png) [@wangxr1985](https://discuss.elastic.co/u/wangxr1985)\ **Post date:** [July 20, 2019, 4:09am UTC](https://discuss.elastic.co/t/can-i-use-gzip-or-some-other-compression-algorithms-when-i-create-snapshots-to-hdfs/188600/10 "2019-07-20T04:09:25Z")
I know how to compress files on shared FS, but now I need to save data on HDFS and I don't know how to compress files on HDFS. I just know data can be compressed while writting to HDFS by setting compression arguments, like compression =\> "gzip" of logstash webhdfs-output-plugin. ES repository-hdfs does not support arguments like that. ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [July 20, 2019, 5:12am UTC](https://discuss.elastic.co/t/can-i-use-gzip-or-some-other-compression-algorithms-when-i-create-snapshots-to-hdfs/188600/11 "2019-07-20T05:12:12Z")
I just wanted you to test what would be the size difference between a snapshot not compressed by gzip and a snapshot compressed by gzip. To see effectively if there is a significant reduction in which case that could be an acceptable feature request. If the win is minimal, then it's probably not worth it to implement such a thing. ---
**Author:** ![wangxr1985](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/wangxr1985/32/117798_2.png) [@wangxr1985](https://discuss.elastic.co/u/wangxr1985)\ **Post date:** [July 20, 2019, 5:22am UTC](https://discuss.elastic.co/t/can-i-use-gzip-or-some-other-compression-algorithms-when-i-create-snapshots-to-hdfs/188600/12 "2019-07-20T05:22:42Z")
The ES index size equals to the size of its snapshot if the snapshot is not compressed. If I use gzip to compress the snapshot, the size will reduce to 15%. ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [July 20, 2019, 9:16am UTC](https://discuss.elastic.co/t/can-i-use-gzip-or-some-other-compression-algorithms-when-i-create-snapshots-to-hdfs/188600/13 "2019-07-20T09:16:57Z")
I'm not sure it's worth to implement but you can ask for a feature request. Note that I think that you can't compress all files together but one by one (segments by segments may be) to see what could be the exact compression ratio. BTW did you run your test on indices rebuilt with `best_compression` codec? ---
**Author:** ![wangxr1985](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/wangxr1985/32/117798_2.png) [@wangxr1985](https://discuss.elastic.co/u/wangxr1985)\ **Post date:** [July 20, 2019, 10:44am UTC](https://discuss.elastic.co/t/can-i-use-gzip-or-some-other-compression-algorithms-when-i-create-snapshots-to-hdfs/188600/14 "2019-07-20T10:44:51Z")
best\_compression codec setting will reduce the index size to 88%, still very large. ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [July 20, 2019, 11:15am UTC](https://discuss.elastic.co/t/can-i-use-gzip-or-some-other-compression-algorithms-when-i-create-snapshots-to-hdfs/188600/15 "2019-07-20T11:15:08Z")
So you are saying that you did the following: - Create an index with `best_compression` - Index documents - Merged the segments to have only one - Run a snapshot on shared FS - Compress each individual file one by one with GZIP And you went from **xxx gb** to **xxx \* 0.15 gb**. Is that what you did exactly? Could you share the output of `du -s` in both cases? ---
**Author:** ![wangxr1985](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/wangxr1985/32/117798_2.png) [@wangxr1985](https://discuss.elastic.co/u/wangxr1985)\ **Post date:** [July 21, 2019, 5:43am UTC](https://discuss.elastic.co/t/can-i-use-gzip-or-some-other-compression-algorithms-when-i-create-snapshots-to-hdfs/188600/16 "2019-07-21T05:43:37Z")
I use logstash to import data from ES index(without best\_compression) to HDFS. Use compression =\> "gzip" in the logstash webhdfs output plugin, output to one .gz file. This is the data size on ES ![image](https://us1.discourse-cdn.com/elastic/original/3X/b/f/bf67b5d967bb0097f6f3caa27520c3b82b49121f.png) This is the data size on HDFS (the first with gzip compression and the second no compression) ![image](https://us1.discourse-cdn.com/elastic/original/3X/f/1/f1f96aa2a2f91510326696211c4ed4712c502ea9.png) ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [July 21, 2019, 2:18pm UTC](https://discuss.elastic.co/t/can-i-use-gzip-or-some-other-compression-algorithms-when-i-create-snapshots-to-hdfs/188600/17 "2019-07-21T14:18:07Z")
But the question is how elasticsearch shards will be compressed not how the source will be compressed. ---
**Author:** ![wangxr1985](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/wangxr1985/32/117798_2.png) [@wangxr1985](https://discuss.elastic.co/u/wangxr1985)\ **Post date:** [July 21, 2019, 4:07pm UTC](https://discuss.elastic.co/t/can-i-use-gzip-or-some-other-compression-algorithms-when-i-create-snapshots-to-hdfs/188600/18 "2019-07-21T16:07:28Z")
My question is how to compress snapshot backup data on HDFS. I know I can use logstash to import data from ES to HDFS with gzip argument, or take a snapshot to HDFS and then compress files on HDFS with hadoop API. But both of them are not graceful. ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [July 21, 2019, 5:36pm UTC](https://discuss.elastic.co/t/can-i-use-gzip-or-some-other-compression-algorithms-when-i-create-snapshots-to-hdfs/188600/19 "2019-07-21T17:36:46Z")
Today there is no graceful way to what you ask, and I think this is because we don't expect significant savings from gzip compression of snapshots: the data that makes up a snapshot is already compressed. However I do not know of recent validation of this expectation. If you have evidence that it's worth compressing snapshots further then it may be worth us investigating this idea and maybe adding this feature. However the results you've shown so far don't help. Your 85% compression ratio is for compressing the JSON source, it seems, and that tells us nothing about the effect of compression on snapshots. ---
**Author:** ![wangxr1985](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/wangxr1985/32/117798_2.png) [@wangxr1985](https://discuss.elastic.co/u/wangxr1985)\ **Post date:** [July 22, 2019, 8:47am UTC](https://discuss.elastic.co/t/can-i-use-gzip-or-some-other-compression-algorithms-when-i-create-snapshots-to-hdfs/188600/20 "2019-07-22T08:47:10Z")
> [@DavidTurner](#): > > the data that makes up a snapshot is already compressed. But my test result is: index size = snapshot size = json text file dumped by elasticdump [Next page](https://discuss.elastic.co/t/can-i-use-gzip-or-some-other-compression-algorithms-when-i-create-snapshots-to-hdfs/188600.md?page=2)