# No index compression with "best\_compression" in 6.3.2 **URL:** \ **Category:** Elasticsearch\ **Created:** [September 19, 2018, 3:04am UTC](https://discuss.elastic.co/t/no-index-compression-with-best-compression-in-6-3-2/149046 "2018-09-19T03:04:15Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![Hari\_Prasad](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hari_prasad/32/56784_2.png) [@Hari\_Prasad](https://discuss.elastic.co/u/Hari_Prasad)\ **Post date:** [September 19, 2018, 3:04am UTC](https://discuss.elastic.co/t/no-index-compression-with-best-compression-in-6-3-2/149046/1 "2018-09-19T03:04:16Z")
I am using Elasticsearch 6.3.2 to index log data, which is sent from Logstash 6.3.2. Below is my Logstash pipeline config (output part) output { elasticsearch { hosts =\> "my-host:9200" index =\> "%{[@metadata][index\_type]}-%{+YYYY.MM.dd}" } } I am using rest call to set the index compression in Elasticsearch with the below one PUT log-2018.08.30 { "settings" : { "index" : { "number\_of\_shards" : 64, "number\_of\_replicas" : 2, "codec":"best\_compression" } } } But with this config, I am not able to achieve any index compression, rather it is bloating.That is for a log file of size less than 2 GB the index size come around 2 GB. Kindly help me with understanding why there are no compression or what is the mistake that i have done the in the above setup. Thanks ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [September 19, 2018, 5:24am UTC](https://discuss.elastic.co/t/no-index-compression-with-best-compression-in-6-3-2/149046/2 "2018-09-19T05:24:23Z")
The best way to enable best\_compression is to add it to an [index template](https://www.elastic.co/guide/en/elasticsearch/reference/6.4/indices-templates.html). This will make it apply to all new indices that the template applies to. ---
**Author:** ![Hari\_Prasad](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hari_prasad/32/56784_2.png) [@Hari\_Prasad](https://discuss.elastic.co/u/Hari_Prasad)\ **Post date:** [September 19, 2018, 6:37am UTC](https://discuss.elastic.co/t/no-index-compression-with-best-compression-in-6-3-2/149046/3 "2018-09-19T06:37:02Z")
Thank you for the suggestion @Christian_Dahlqvist. but is there any mistake in my current config. When i get the index settings i am able see that the codec is applied to the index. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [September 19, 2018, 6:53am UTC](https://discuss.elastic.co/t/no-index-compression-with-best-compression-in-6-3-2/149046/4 "2018-09-19T06:53:41Z")
If you see it applied in the index settings it is applied. The size your data take up on disk [will largely depend on how much enrichment you do and how optimised your mappings are](https://www.elastic.co/blog/filebeat-modiles-access-logs-and-elasticsearch-storage-requirements). The improved compression applies to the source and usually in my experience gives a 10%-20% space saving as the data is compressed by default. ---
**Author:** ![Hari\_Prasad](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hari_prasad/32/56784_2.png) [@Hari\_Prasad](https://discuss.elastic.co/u/Hari_Prasad)\ **Post date:** [September 20, 2018, 6:39am UTC](https://discuss.elastic.co/t/no-index-compression-with-best-compression-in-6-3-2/149046/5 "2018-09-20T06:39:34Z")
The mapping I use is as below { "log\_instance-2018.07.31": { "mappings": { "doc": { "properties": { "@timestamp": { "type": "date" }, "@version": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore\_above": 256 } } }, "hostname": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore\_above": 256 } } }, "message": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore\_above": 256 } } }, "source": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore\_above": 256 } } }, "thread": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore\_above": 256 } } } } } } } } @Christian_Dahlqvist: Does this have any problem that would hamper the compression? Or is there any other settings that i will have to tweak to achieve compression ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [September 20, 2018, 7:05am UTC](https://discuss.elastic.co/t/no-index-compression-with-best-compression-in-6-3-2/149046/6 "2018-09-20T07:05:05Z")
I see that you seem to be using the default dynamic mappings. These do index every field other as text and keyword, which adds a lot of flexibility but can also take up quite a bit of extra space on disk. I would recommend you go through your mappings and optimize them according to [these guidelines](https://www.elastic.co/guide/en/elasticsearch/reference/6.4/tune-for-disk-usage.html). The `best_compression` codec applies to the JSON source, so these mappings will not be affected. ---
**Author:** ![Hari\_Prasad](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hari_prasad/32/56784_2.png) [@Hari\_Prasad](https://discuss.elastic.co/u/Hari_Prasad)\ **Post date:** [September 20, 2018, 9:37am UTC](https://discuss.elastic.co/t/no-index-compression-with-best-compression-in-6-3-2/149046/7 "2018-09-20T09:37:54Z")
@Christian_Dahlqvist: thank you. I will try to follow the guidelines to optimise the disk usage. Do you have any suggestion or link which says exactly how to correctly set the codec ? will setting the codec via template make sure it is applied? ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [October 18, 2018, 9:38am UTC](https://discuss.elastic.co/t/no-index-compression-with-best-compression-in-6-3-2/149046/8 "2018-10-18T09:38:05Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.