# How to Reduce Received Logs Size in ELK Stack?
**URL:** https://discuss.elastic.co/t/how-to-reduce-received-logs-size-in-elk-stack/301066
**Category:** Elasticsearch
**Created:** [March 30, 2022, 9:39am UTC](https://discuss.elastic.co/t/how-to-reduce-received-logs-size-in-elk-stack/301066 "2022-03-30T09:39:04Z")
**Posts on this page:** 20
**Page:** 1
### Author:  [@wcpoon](https://discuss.elastic.co/u/wcpoon)
#### Post date: [March 30, 2022, 9:39am UTC](https://discuss.elastic.co/t/how-to-reduce-received-logs-size-in-elk-stack/301066/1 "2022-03-30T09:39:04Z")
Hi,
I would like to ask, any way we can compress / dedup the sizes of the logs that we received in ELK stack?
Thanks.
---
### Author:  [@wcpoon](https://discuss.elastic.co/u/wcpoon)
#### Post date: [March 31, 2022, 2:17am UTC](https://discuss.elastic.co/t/how-to-reduce-received-logs-size-in-elk-stack/301066/2 "2022-03-31T02:17:43Z")
Is there anyone can give some advice on this?
---
### Author:  [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)
#### Post date: [March 31, 2022, 2:27am UTC](https://discuss.elastic.co/t/how-to-reduce-received-logs-size-in-elk-stack/301066/3 "2022-03-31T02:27:10Z")
Have you checked this part of the documentation about [tunning for disk usage](https://www.elastic.co/guide/en/elasticsearch/reference/current/tune-for-disk-usage.html#tune-for-disk-usage)?
Those are the some things that you can do to help you reduce the indices sizes.
---
### Author:  [@wcpoon](https://discuss.elastic.co/u/wcpoon)
#### Post date: [March 31, 2022, 3:05am UTC](https://discuss.elastic.co/t/how-to-reduce-received-logs-size-in-elk-stack/301066/4 "2022-03-31T03:05:11Z")
Hi,
currently my indices with 1 x Primary Shard & 1 x Replica..
I can consider to Shrink & Force Merge it after 60 days in WARM phase..
If my Shrink set as 1 shard count, is it my indices will reduce from 1 x Primary, 1 x Replica to 1 Primary Shard only?
and How about Force Merge?
Hope to get some clarification from you, I still new to Elasticsearch, appreciate for your help.
Thanks.
---
### Author:  [@warkolm](https://discuss.elastic.co/u/warkolm)
#### Post date: [March 31, 2022, 3:38am UTC](https://discuss.elastic.co/t/how-to-reduce-received-logs-size-in-elk-stack/301066/5 "2022-03-31T03:38:54Z")
Just a heads up that while we endeavour to reply to every thread, we don't provide SLAs on requests 🙂
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [March 31, 2022, 3:59am UTC](https://discuss.elastic.co/t/how-to-reduce-received-logs-size-in-elk-stack/301066/6 "2022-03-31T03:59:41Z")
Shrinking an ondex reduces the number of primary shsrds, so will not have any effect in your case. You can reduce the number of replicas but that will reduce availability and resiliency. Forcemerging may reduce size if you at this time apply best compression but may otherwise not necessarily shrink the size much. I would recommend looking atvthe link provided and make sure your mappings are optimized.
---
### Author:  [@d.silwon](https://discuss.elastic.co/u/d.silwon)
#### Post date: [March 31, 2022, 5:24am UTC](https://discuss.elastic.co/t/how-to-reduce-received-logs-size-in-elk-stack/301066/7 "2022-03-31T05:24:20Z")
Are you using compression? How do you send logs to ELK?
Are you using filebeat? If yes then you can exclude some unnecessary fields from each event for the help of parameter drop\_fields in filebeat configuration.
---
### Author:  [@wcpoon](https://discuss.elastic.co/u/wcpoon)
#### Post date: [March 31, 2022, 5:43am UTC](https://discuss.elastic.co/t/how-to-reduce-received-logs-size-in-elk-stack/301066/8 "2022-03-31T05:43:58Z")
Hi,
I'm sending syslog to logstash.
Not using any beats yet...
---
### Author:  [@d.silwon](https://discuss.elastic.co/u/d.silwon)
#### Post date: [March 31, 2022, 5:50am UTC](https://discuss.elastic.co/t/how-to-reduce-received-logs-size-in-elk-stack/301066/9 "2022-03-31T05:50:23Z")
What about compression?
---
### Author:  [@wcpoon](https://discuss.elastic.co/u/wcpoon)
#### Post date: [March 31, 2022, 5:54am UTC](https://discuss.elastic.co/t/how-to-reduce-received-logs-size-in-elk-stack/301066/10 "2022-03-31T05:54:22Z")
I'm using default compression.
---
### Author:  [@d.silwon](https://discuss.elastic.co/u/d.silwon)
#### Post date: [March 31, 2022, 6:19am UTC](https://discuss.elastic.co/t/how-to-reduce-received-logs-size-in-elk-stack/301066/11 "2022-03-31T06:19:46Z")
Please change it to best\_compression codec.
---
### Author:  [@wcpoon](https://discuss.elastic.co/u/wcpoon)
#### Post date: [March 31, 2022, 6:25am UTC](https://discuss.elastic.co/t/how-to-reduce-received-logs-size-in-elk-stack/301066/12 "2022-03-31T06:25:40Z")
Hi d.silwon,
Sorry, i'm new to Elasticsearch, may I know, which config file I should configure for the index.codec to best\_compression? or how to change from default to best\_compression?
Thanks.
---
### Author:  [@d.silwon](https://discuss.elastic.co/u/d.silwon)
#### Post date: [March 31, 2022, 6:31am UTC](https://discuss.elastic.co/t/how-to-reduce-received-logs-size-in-elk-stack/301066/13 "2022-03-31T06:31:04Z")
It can be change in the template of the your index. Here you are the example:
```auto
{
"index": {
"codec": "best_compression"
}
}
```
---
### Author:  [@wcpoon](https://discuss.elastic.co/u/wcpoon)
#### Post date: [March 31, 2022, 6:42am UTC](https://discuss.elastic.co/t/how-to-reduce-received-logs-size-in-elk-stack/301066/14 "2022-03-31T06:42:27Z")
this is my current index templates,
```auto
{
"index_templates" : [
{
"name" : "logstash",
"index_template" : {
"index_patterns" : [
"logstash-*"
],
"template" : {
"settings" : {
"index" : {
"lifecycle" : {
"name" : "logstash-policy",
"rollover_alias" : "logstash"
},
"number_of_shards" : "1",
"refresh_interval" : "5s"
}
},
"mappings" : {
"dynamic_templates" : [
{
"message_field" : {
"path_match" : "message",
"mapping" : {
"norms" : false,
"type" : "text"
},
"match_mapping_type" : "string"
}
},
{
"string_fields" : {
"mapping" : {
"norms" : false,
"type" : "text",
"fields" : {
"keyword" : {
"ignore_above" : 256,
"type" : "keyword"
}
}
},
"match_mapping_type" : "string",
"match" : "*"
}
}
],
"properties" : {
"@timestamp" : {
"type" : "date"
},
"geoip" : {
"dynamic" : true,
"properties" : {
"ip" : {
"type" : "ip"
},
"latitude" : {
"type" : "half_float"
},
"location" : {
"type" : "geo_point"
},
"longitude" : {
"type" : "half_float"
}
}
},
"@version" : {
"type" : "keyword"
}
}
}
},
"composed_of" : [],
"priority" : 200,
"version" : 80001,
"_meta" : {
"description" : "index template for logstash-output-elasticsearch"
}
}
}
]
}
```
I'm getting this error,
```auto
PUT _index_template/logstash
{
"index": {
"codec": "best_compression"
}
}
```
```auto
{
"error" : {
"root_cause" : [
{
"type" : "x_content_parse_exception",
"reason" : "[2:3] [index_template] unknown field [index]"
}
],
"type" : "x_content_parse_exception",
"reason" : "[2:3] [index_template] unknown field [index]"
},
"status" : 400
}
```
---
### Author:  [@wcpoon](https://discuss.elastic.co/u/wcpoon)
#### Post date: [March 31, 2022, 7:06am UTC](https://discuss.elastic.co/t/how-to-reduce-received-logs-size-in-elk-stack/301066/15 "2022-03-31T07:06:50Z")
Can I set
`index.codec: best_compression` in Elasticsearch.yml ?
---
### Author:  [@d.silwon](https://discuss.elastic.co/u/d.silwon)
#### Post date: [March 31, 2022, 7:11am UTC](https://discuss.elastic.co/t/how-to-reduce-received-logs-size-in-elk-stack/301066/16 "2022-03-31T07:11:36Z")
Here you are example of index template API in case of my example template called applications:
```auto
PUT _index_template/applications
{
"index_patterns": ["applications-*"],
"template": {
"settings": {
"codec": "best_compression"
}
}
}
```
Please adjust it to your needs.
---
### Author:  [@wcpoon](https://discuss.elastic.co/u/wcpoon)
#### Post date: [March 31, 2022, 8:11am UTC](https://discuss.elastic.co/t/how-to-reduce-received-logs-size-in-elk-stack/301066/17 "2022-03-31T08:11:27Z")
after changed, it looks like this,
GET \_index\_template/logstash
```auto
{
"index_templates" : [
{
"name" : "logstash",
"index_template" : {
"index_patterns" : [
"logstash-*"
],
"template" : {
"settings" : {
"index" : {
"lifecycle" : {
"name" : "logstash-policy",
"rollover_alias" : "logstash"
},
"codec" : "best_compression",
"refresh_interval" : "5s",
"number_of_shards" : "1"
}
},
"mappings" : {
"dynamic_templates" : [
{
"message_field" : {
"path_match" : "message",
"mapping" : {
"norms" : false,
"type" : "text"
},
"match_mapping_type" : "string"
}
},
{
"string_fields" : {
"mapping" : {
"norms" : false,
"type" : "text",
"fields" : {
"keyword" : {
"ignore_above" : 256,
"type" : "keyword"
}
}
},
"match_mapping_type" : "string",
"match" : "*"
}
}
],
"properties" : {
"@timestamp" : {
"type" : "date"
},
"geoip" : {
"dynamic" : true,
"properties" : {
"ip" : {
"type" : "ip"
},
"latitude" : {
"type" : "half_float"
},
"location" : {
"type" : "geo_point"
},
"longitude" : {
"type" : "half_float"
}
}
},
"@version" : {
"type" : "keyword"
}
}
}
},
"composed_of" : [],
"priority" : 200,
"version" : 80001,
"_meta" : {
"description" : "index template for logstash-output-elasticsearch"
}
}
}
]
}
```
The config Is it looks ok?
When the codec will take affect?
---
### Author:  [@d.silwon](https://discuss.elastic.co/u/d.silwon)
#### Post date: [March 31, 2022, 8:14am UTC](https://discuss.elastic.co/t/how-to-reduce-received-logs-size-in-elk-stack/301066/18 "2022-03-31T08:14:14Z")
This change will be applied for new indexes, during the creation. In case of existing index you should try change settings for each index separately.
---
### Author:  [@wcpoon](https://discuss.elastic.co/u/wcpoon)
#### Post date: [March 31, 2022, 8:18am UTC](https://discuss.elastic.co/t/how-to-reduce-received-logs-size-in-elk-stack/301066/19 "2022-03-31T08:18:57Z")
But I already changed the index templates (logstash).
It will take effect for new indexes, Am I right?
---
### Author:  [@d.silwon](https://discuss.elastic.co/u/d.silwon)
#### Post date: [March 31, 2022, 8:37am UTC](https://discuss.elastic.co/t/how-to-reduce-received-logs-size-in-elk-stack/301066/20 "2022-03-31T08:37:56Z")
You are right
[Next page](https://discuss.elastic.co/t/how-to-reduce-received-logs-size-in-elk-stack/301066.md?page=2)