# Raw logs to data compression ratio **URL:** \ **Category:** Elasticsearch\ **Created:** [May 24, 2022, 2:43am UTC](https://discuss.elastic.co/t/raw-logs-to-data-compression-ratio/305440 "2022-05-24T02:43:37Z")\ **Posts on this page:** 18\ **Page:** 1
**Author:** ![michaelv](https://avatars.discourse-cdn.com/v4/letter/m/90db22/32.png) [@michaelv](https://discuss.elastic.co/u/michaelv)\ **Post date:** [May 24, 2022, 2:43am UTC](https://discuss.elastic.co/t/raw-logs-to-data-compression-ratio/305440/1 "2022-05-24T02:43:37Z")
Hi All, I've been investigating some compression ratios of my live logs. Two main catergories. Auditd logs and other logs like /var/log/messages etc. Both being ingested via filebeat 7.11 and running 7.11 cluster. So what I've found is (on an average day, after dividing over a 7 day period). Auditd raw logs 3.97G Index size 13.27G Ratio about 3.34 Other raw logs 3.61G Index size 10G Ratio 2.77 These seems abnormally high ration from raw logs to index sizes. In many places, people talk about 30%-100% of raw logs size, I'm getting around 200% of raw log size. Question: 1. Can the overhead be reduced. (Overhead being the additional data like filebeat version, source IP, source hostname etc etc.. ) 2. I assume its already using deflate as a compression mode since its 7.11. Are there ways to improve compression ratios. Regards, Michael ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [May 24, 2022, 2:44am UTC](https://discuss.elastic.co/t/raw-logs-to-data-compression-ratio/305440/2 "2022-05-24T02:44:32Z")
What do the mappings for these look like, have you spent time working on them? ---
**Author:** ![michaelv](https://avatars.discourse-cdn.com/v4/letter/m/90db22/32.png) [@michaelv](https://discuss.elastic.co/u/michaelv)\ **Post date:** [May 24, 2022, 2:57am UTC](https://discuss.elastic.co/t/raw-logs-to-data-compression-ratio/305440/3 "2022-05-24T02:57:00Z")
Hi Mark, I'm using standard filebeat index. The /var/log/messages goes into standard filebeat-7.\* index. The auditbeat uses the filebeat-auditd-7.\* otherwise the same filebeat standard. Regards, Michael ---
**Author:** ![michaelv](https://avatars.discourse-cdn.com/v4/letter/m/90db22/32.png) [@michaelv](https://discuss.elastic.co/u/michaelv)\ **Post date:** [May 24, 2022, 2:59am UTC](https://discuss.elastic.co/t/raw-logs-to-data-compression-ratio/305440/4 "2022-05-24T02:59:28Z")
I've sort of inherited support of this. I'm not really sure was any testing done to optimize it. I understand you can choose to index or not index certain fields. Though I'm not sure if that will help. Its not like these logs have a huge range of field types. ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [May 24, 2022, 3:02am UTC](https://discuss.elastic.co/t/raw-logs-to-data-compression-ratio/305440/5 "2022-05-24T03:02:24Z")
The default mappings are not super efficient, so you should really take a look at them. ---
**Author:** ![michaelv](https://avatars.discourse-cdn.com/v4/letter/m/90db22/32.png) [@michaelv](https://discuss.elastic.co/u/michaelv)\ **Post date:** [May 24, 2022, 3:19am UTC](https://discuss.elastic.co/t/raw-logs-to-data-compression-ratio/305440/6 "2022-05-24T03:19:42Z")
This implementation has been running for more than a year. Change index mappings, can it be done on live indexes? Or must the index be trashed and recreated? Also setting best\_compression, can not be done on running indexes. Is it possible to be set, on next created index in sequence? ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [May 24, 2022, 3:23am UTC](https://discuss.elastic.co/t/raw-logs-to-data-compression-ratio/305440/7 "2022-05-24T03:23:46Z")
Take a look at index templates (or composable templates at they are known now). Basically the idea is you set the template for the next set of indices that are created by the Beats, and they define the mappings and compression method. Saves manually doing it. ---
**Author:** ![michaelv](https://avatars.discourse-cdn.com/v4/letter/m/90db22/32.png) [@michaelv](https://discuss.elastic.co/u/michaelv)\ **Post date:** [May 24, 2022, 3:47am UTC](https://discuss.elastic.co/t/raw-logs-to-data-compression-ratio/305440/8 "2022-05-24T03:47:57Z")
Okay I see what you mean. If I put "index.codec: best\_compression" in Elasticsearch.yml from the recommendation from [Part 2.0: The true story behind Elasticsearch storage requirements | Elastic Blog](https://www.elastic.co/blog/elasticsearch-storage-the-true-story-2.0) Would that effect all newly created indexes? Regards, Michael ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [May 24, 2022, 3:49am UTC](https://discuss.elastic.co/t/raw-logs-to-data-compression-ratio/305440/9 "2022-05-24T03:49:54Z")
If it's in an appropriate template, yes. ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [May 24, 2022, 4:47am UTC](https://discuss.elastic.co/t/raw-logs-to-data-compression-ratio/305440/10 "2022-05-24T04:47:37Z")
Also @michaelv Most people use Raw Logs to Primary storage when referring to Ratio, not total index size which includes replica,,, how many replicas do you have? BTW that is a pretty old blog there is a slightly newer one [here](https://www.elastic.co/blog/filebeat-modiles-access-logs-and-elasticsearch-storage-requirements) Here are some of the latest > **[Tune for disk usage | Elasticsearch Guide \[8.2\] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/tune-for-disk-usage.html)** ---
**Author:** ![michaelv](https://avatars.discourse-cdn.com/v4/letter/m/90db22/32.png) [@michaelv](https://discuss.elastic.co/u/michaelv)\ **Post date:** [May 24, 2022, 5:17am UTC](https://discuss.elastic.co/t/raw-logs-to-data-compression-ratio/305440/11 "2022-05-24T05:17:35Z")
HI Stephen, 1 replica only. The blog post was given by ELK support staff. I'll read the newer blog post and see what its about. Regards, Michael ---
**Author:** ![michaelv](https://avatars.discourse-cdn.com/v4/letter/m/90db22/32.png) [@michaelv](https://discuss.elastic.co/u/michaelv)\ **Post date:** [May 24, 2022, 6:25am UTC](https://discuss.elastic.co/t/raw-logs-to-data-compression-ratio/305440/12 "2022-05-24T06:25:27Z")
One odd questin. Using index.codec: best\_compression I thought 7.10 by default uses deflate as the compression. Why would you need to hardcode the best\_compression? I've tried modifying the template, and yes the new index takes the best\_compression index. Regards, Michael ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [May 24, 2022, 6:43am UTC](https://discuss.elastic.co/t/raw-logs-to-data-compression-ratio/305440/13 "2022-05-24T06:43:34Z")
> [@michaelv](#): > > I thought 7.10 by default uses deflate as the compression. No, it does not. > [@michaelv](#): > > Why would you need to hardcode the best\_compression? Best compression is not the default as it adds a significant amount of overhead at indexing time. ---
**Author:** ![michaelv](https://avatars.discourse-cdn.com/v4/letter/m/90db22/32.png) [@michaelv](https://discuss.elastic.co/u/michaelv)\ **Post date:** [May 24, 2022, 8:27am UTC](https://discuss.elastic.co/t/raw-logs-to-data-compression-ratio/305440/14 "2022-05-24T08:27:39Z")
Thanks Christian.. they are importing something like 50 million records per day.. not sure having a higher overhead is a good idea. I've read doing warm nodes to do higher compression.. however, they only have 3 nodes. So that is not really an option as its the same node for the hot indicies. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [May 24, 2022, 9:19am UTC](https://discuss.elastic.co/t/raw-logs-to-data-compression-ratio/305440/15 "2022-05-24T09:19:34Z")
For lower data volumes the overhead of tbe best compression codec may be irrelevant. ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [May 25, 2022, 8:07pm UTC](https://discuss.elastic.co/t/raw-logs-to-data-compression-ratio/305440/16 "2022-05-25T20:07:58Z")
50M records / day ~600 Records / sec Avg Certainly in the realm / value of best compression... I have use cases 10-20x+ using best compression Basically it is a TCO and performance equation. You might need to trade /add a couple % of CPU etc for greater storage savings. I would say test and measure. So what I was kinda saying before if your numbers are for total storage ... most people refer to primary storage when they speak of ratio > Auditd raw logs 3.97G > Index size 13.27G \