# Issue with Compression Not Working in Elastic Agent

**URL:** <https://discuss.elastic.co/t/issue-with-compression-not-working-in-elastic-agent/358343>\
**Category:** Elastic Search\
**Created:** [April 27, 2024, 2:00pm UTC](https://discuss.elastic.co/t/issue-with-compression-not-working-in-elastic-agent/358343 "2024-04-27T14:00:39Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![wangsubo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/wangsubo/32/133865_2.png) [@wangsubo](https://discuss.elastic.co/u/wangsubo)\
**Post date:** [April 27, 2024, 2:00pm UTC](https://discuss.elastic.co/t/issue-with-compression-not-working-in-elastic-agent/358343/1 "2024-04-27T14:00:39Z")

</div>

I am using Elastic Agent to collect data from Fortigate. In order to reduce storage space, I have set compression\_level: 5 in elastic-agent.yml.

outputs:  
default:  
type: elasticsearch  
hosts: [xxx.xxx.xxx.xxx:9200]  
api\_key: "example-key"  
#username: "elastic"  
#password: "changeme"  
compression\_level: 5

Additionally, I have configured settings in Fleet -\> Settings -\> Outputs -\> Advanced YAML configuration.

```
  bulk_max_size: 4096
  worker: 1
  queue.mem.events: 8192
  queue.mem.flush.min_events: 4096
  queue.mem.flush.timeout: 5s
  compression_level: 5
  connection_idle_timeout: 3s

```

When I check the index stats, both size\_in\_bytes and total\_data\_set\_size\_in\_bytes are the same, indicating that the compression has not been successful. Can anyone suggest what might be wrong with my configuration? Thank you.

```
  "store": {
    "size_in_bytes": 304600,
    "total_data_set_size_in_bytes": 304600,
    "reserved_in_bytes": 0
  },

```

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [April 27, 2024, 2:31pm UTC](https://discuss.elastic.co/t/issue-with-compression-not-working-in-elastic-agent/358343/2 "2024-04-27T14:31:21Z")

</div>

Those are different things.

The compression configuration in the Elastic Agent that you mentioned is related to the compression of the request that will be sent to Elasticsearch, it has no influence in the compression of the data in Elasticsearch itself after it is indexed.

The compressions of the data in Elasticsearch is managed by a setting on the index template named `index.codec`, there are 2 options `default` and `best_compression`.

Since you are using Elastic Agent and its integrations, the compression is already set to `best_compression`.

---

<div class="post-metadata">

**Author:** ![wangsubo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/wangsubo/32/133865_2.png) [@wangsubo](https://discuss.elastic.co/u/wangsubo)\
**Post date:** [April 27, 2024, 3:01pm UTC](https://discuss.elastic.co/t/issue-with-compression-not-working-in-elastic-agent/358343/3 "2024-04-27T15:01:07Z")

</div>

Thank you for your response. I have checked the index template and confirmed that it is set to `best_compression`. However, the values for both `size_in_bytes` and `total_data_set_size_in_bytes` remain the same.

How can I determine whether the compression has been successful or not?

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [April 27, 2024, 3:16pm UTC](https://discuss.elastic.co/t/issue-with-compression-not-working-in-elastic-agent/358343/4 "2024-04-27T15:16:08Z")

</div>

Not sure what you want to determine, if the index codec is set the `best_compression` then the stored data is already compressed.

If I'm not wrong the value you see in `size_in_bytes` is after the compression, is the value used by the underlying data in disk.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 25, 2024, 3:16pm UTC](https://discuss.elastic.co/t/issue-with-compression-not-working-in-elastic-agent/358343/5 "2024-05-25T15:16:47Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.