Elasticsearch: Snapshot repository - index compression - "hard" archiving

Elastic Stack Elasticsearch
1

Hi!

Are the snapshots taken within elasticsearch repositories compressed in some fairly efficient way by default?

When you "permanently" secure (e.g. BD-ROM) a snapshot repository, is it worth compressing it to some archive (rar, zip, 7z, etc.)?

Does such compression make absolutely no sense and won't give any real size reduction?
My guess is that it might be...

The extra compression is also a very large computational cost.

Is it worth it here to limit myself to a simple tar (tarball) with file splits of specific sizes?

Thanks for your help!

2

Elasticsearch (via Lucene) compresses all the data it writes to disk, but snapshots are not compressed any further than regular indices. I've heard anecdotal reports that some additional savings are possible in some circumstances.

Be aware that compressed data can be much harder to recover if it suffers even a tiny amount of corruption, which is a big risk in long term archival. See e.g. this article:

https://www.nongnu.org/lzip/xz_inadequate.html

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.