# How to Reduce Received Logs Size in ELK Stack? **URL:** \ **Category:** Elasticsearch\ **Created:** [March 30, 2022, 9:39am UTC](https://discuss.elastic.co/t/how-to-reduce-received-logs-size-in-elk-stack/301066 "2022-03-30T09:39:04Z")\ **Posts on this page:** 13\ **Page:** 2
**Author:** ![wcpoon](https://avatars.discourse-cdn.com/v4/letter/w/da6949/32.png) [@wcpoon](https://discuss.elastic.co/u/wcpoon)\ **Post date:** [April 1, 2022, 7:37am UTC](https://discuss.elastic.co/t/how-to-reduce-received-logs-size-in-elk-stack/301066/21 "2022-04-01T07:37:25Z")
Hi @d.silwon , my new index is apply to new index template today, and it show `codec: best_compression` in indices settings. Any idea how to verify the `codec: best_compression` is working? ---
**Author:** ![d.silwon](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/d.silwon/32/65853_2.png) [@d.silwon](https://discuss.elastic.co/u/d.silwon)\ **Post date:** [April 1, 2022, 9:33am UTC](https://discuss.elastic.co/t/how-to-reduce-received-logs-size-in-elk-stack/301066/22 "2022-04-01T09:33:45Z")
Hi @wcpoon, Good question 🙂 First of all check new index settings: GET /index\_name-2022.04.01/\_settings Maybe enought will be to compare index size between few days: GET /\_cat/indices/index\_name-2022.\*?v&s=index No more idea right now. Best Regards, Dan ---
**Author:** ![wcpoon](https://avatars.discourse-cdn.com/v4/letter/w/da6949/32.png) [@wcpoon](https://discuss.elastic.co/u/wcpoon)\ **Post date:** [April 4, 2022, 5:33am UTC](https://discuss.elastic.co/t/how-to-reduce-received-logs-size-in-elk-stack/301066/23 "2022-04-04T05:33:58Z")
Hi @d.silwon After few days of running in best\_compression. ```auto health status index uuid pri rep docs.count docs.deleted store.size pri.store.size green open logstash-2022.04.01-000004-000003 DbksMBHERza3gISPqMAmlA 1 1 114920759 0 60.6gb 30.3gb green open logstash-2022.04.01-000004-000004 QnEcFMmNSdKuRPejabhH5g 1 1 83416347 0 44.6gb 22.2gb green open logstash-2022.04.01-000004-2 A4ALCm71Q-6Bq2Rv2BGAUw 1 1 115001745 0 60.8gb 30.3gb ``` ```auto { "logstash-2022.04.01-000004-000003" : { "settings" : { "index" : { "lifecycle" : { "name" : "logstash-policy", "rollover_alias" : "logstash", "indexing_complete" : "true" }, "codec" : "best_compression", "routing" : { "allocation" : { "include" : { "_tier_preference" : "data_content" } } }, "refresh_interval" : "5s", "number_of_shards" : "1", "provided_name" : "logstash-2022.04.01-000004-000003", "creation_date" : "1648894887761", "number_of_replicas" : "1", "uuid" : "DbksMBHERza3gISPqMAmlA", "version" : { "created" : "8010199" } } } } } ``` It is taking around 60GB++ for 100m of data... Is it normal? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [April 4, 2022, 5:42am UTC](https://discuss.elastic.co/t/how-to-reduce-received-logs-size-in-elk-stack/301066/24 "2022-04-04T05:42:21Z")
It is hard to tell what is normal or not as it depends on the size of the documents as well as the mappings used. The mappings can make a huge difference on the size as it determines how all the fields are indexed and the default mappings are designed for flexibility rather than storage efficiency. I would recommend going through the link provided earlier around tuning for disk usage. ---
**Author:** ![d.silwon](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/d.silwon/32/65853_2.png) [@d.silwon](https://discuss.elastic.co/u/d.silwon)\ **Post date:** [April 4, 2022, 6:01am UTC](https://discuss.elastic.co/t/how-to-reduce-received-logs-size-in-elk-stack/301066/25 "2022-04-04T06:01:17Z")
As @Christian_Dahlqvist wrote it's hard to tell if it is normal or not. Maybe look at a sample document in your index and consider if all the fields are needed. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [April 4, 2022, 6:05am UTC](https://discuss.elastic.co/t/how-to-reduce-received-logs-size-in-elk-stack/301066/26 "2022-04-04T06:05:59Z")
[Here](https://www.elastic.co/blog/filebeat-modiles-access-logs-and-elasticsearch-storage-requirements) is a very old blog post that shows how the size on disk depends on the mappings used and the number of fields indexed. Even though most of the recommendation may no longer be valid as there has been a lot of enhancements and changes over the years the principle still applies, which is described in a more up to date form in the link that was provided. ---
**Author:** ![d.silwon](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/d.silwon/32/65853_2.png) [@d.silwon](https://discuss.elastic.co/u/d.silwon)\ **Post date:** [April 4, 2022, 6:11am UTC](https://discuss.elastic.co/t/how-to-reduce-received-logs-size-in-elk-stack/301066/27 "2022-04-04T06:11:30Z")
@wcpoon what is the size of file/files from whole day which you load to the index logstash-\*? I would like to know this comparison to index in ELK. Thanks ---
**Author:** ![wcpoon](https://avatars.discourse-cdn.com/v4/letter/w/da6949/32.png) [@wcpoon](https://discuss.elastic.co/u/wcpoon)\ **Post date:** [April 4, 2022, 6:24am UTC](https://discuss.elastic.co/t/how-to-reduce-received-logs-size-in-elk-stack/301066/28 "2022-04-04T06:24:18Z")
Hi @d.silwon, I'm pushing my vSphere logs to logstash, input to logstash as syslog and output to Elasticsearch. Don't think so I can direct load to Elasticsearch. ---
**Author:** ![d.silwon](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/d.silwon/32/65853_2.png) [@d.silwon](https://discuss.elastic.co/u/d.silwon)\ **Post date:** [April 4, 2022, 6:26am UTC](https://discuss.elastic.co/t/how-to-reduce-received-logs-size-in-elk-stack/301066/29 "2022-04-04T06:26:33Z")
@wcpoon No, I just want to compare only the source size vs the index size. ---
**Author:** ![wcpoon](https://avatars.discourse-cdn.com/v4/letter/w/da6949/32.png) [@wcpoon](https://discuss.elastic.co/u/wcpoon)\ **Post date:** [April 4, 2022, 6:33am UTC](https://discuss.elastic.co/t/how-to-reduce-received-logs-size-in-elk-stack/301066/30 "2022-04-04T06:33:26Z")
I forward the vSphere logs from Log Insight to ELK stack... In Log Insight, the file size is huge also, but compare to ELK, ELK size is larger than Log Insight.. just wonder anyway I can reduce the size of logs in ELK... ---
**Author:** ![d.silwon](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/d.silwon/32/65853_2.png) [@d.silwon](https://discuss.elastic.co/u/d.silwon)\ **Post date:** [April 4, 2022, 7:01am UTC](https://discuss.elastic.co/t/how-to-reduce-received-logs-size-in-elk-stack/301066/31 "2022-04-04T07:01:16Z")
I think that you should read the blog post mentioned by @Christian_Dahlqvist There are few key things which can help you reduce the size of stored logs. ---
**Author:** ![wcpoon](https://avatars.discourse-cdn.com/v4/letter/w/da6949/32.png) [@wcpoon](https://discuss.elastic.co/u/wcpoon)\ **Post date:** [April 4, 2022, 7:56am UTC](https://discuss.elastic.co/t/how-to-reduce-received-logs-size-in-elk-stack/301066/32 "2022-04-04T07:56:54Z")
Yup, will go through it. Appreciate with your prompt replied. 🙂 ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [May 2, 2022, 7:57am UTC](https://discuss.elastic.co/t/how-to-reduce-received-logs-size-in-elk-stack/301066/33 "2022-05-02T07:57:52Z")
Closed on May 2, 2022, 7:57am UTC This topic was automatically closed 28 days after the last reply. New replies are no longer allowed. [Previous page](https://discuss.elastic.co/t/how-to-reduce-received-logs-size-in-elk-stack/301066.md?page=1)