Changing index.codec on existing index?

Elastic Stack Elasticsearch
1

Hi,

i tried changing the index.codec on a existing index to "best_compression"(DEFLATE) to achieve better compression values, than the standard LZ4.

I closed the index, changed the codec, reopened it and than run the optimize api on it with setting the max_num_segments to 20.

The index did have 998 GB (according to kopf-plugin), after the optimize command finished (took ages, somehow the merging from _optmize only run with 20MB/s) the index still got 996 GB according to kopf.

Is it in the nature of ES that existing indices can not be compressed very good? Can i expect better resuls when setting index.codec at index creating time?

I am tryting to replace the compression of our filesystem with this, since the CPU overhead is quite large now and i think that the performance decrease from ES will not have that big of an impact.

Thanks for your response.

Regards

2

Why not 1?

Sparse values don't compress very well, so it could be that.

3

Why not 1?

I though reducing the number of segments from 943 to 320 would be enough to give me an idea of how the compression is going to work out (maybe it only comes into play if the segments get bigger, so less segments? )

Regardless of that, i am going to try out how the compression performs when setting it at index creation time

4

What kind of data are you storing? Do you have_source enabled?

5

_source is enabled.

We use ES for logging, so the data mostly consist out of netflow data, windows event logs and unix like log files