Hi, I am new to Elasticsearch and I have a question around Elasticsearch compression. I know it uses the codec default and best_compression when creating an index but since my backend storage already provides LZ4 compression, is there away to disable compression at the Elasticsearch Index level? This way I can free up CPU cycles on the ES nodes and let the storage array does all the compression for me.
Thanks in advance.
Thanks, Kathleen. So if someone who doesn't use DAS for storage, then they're stuck with the default settings? I saw some article talking about writing your own codec to disable compression but that requires recompiling ES code base? Is that an accurate statement?
Hi @tommyd, Welcome to the community!
Yes no way AFAIK to disable compression, it is a core part of Elastic.
Unless you are at an extreme scale, Lucene is really, really efficient, it seems like a lot of work and risk and maintenance for perhaps a small gain... especially if you are just getting started with Elastic.
It seems like spending your time on good mappings, index management queries, etc., might be a better use of your time.
Just one opinion.
1 Like
Thanks Stephen. Appreciated for your help and Kathleen's help!
Cheers!
For completeness, yes, you would need to compile and repackage Elasticsearch (and quite possibly some of its dependencies) to disable compression.
I wouldn't be surprised to see worse performance after doing so too - the OS page cache would be working with the uncompressed data and so will fill up much much faster, and this is an extremely important factor for performance. There's a good reason this isn't offered as an option.
1 Like
Thanks, David! This helps a lot.