Issue with Compression Not Working in Elastic Agent

[wangsubo]

I am using Elastic Agent to collect data from Fortigate. In order to reduce storage space, I have set compression_level: 5 in elastic-agent.yml.

outputs:
default:
type: elasticsearch
hosts: [xxx.xxx.xxx.xxx:9200]
api_key: "example-key"
#username: "elastic"
#password: "changeme"
compression_level: 5

Additionally, I have configured settings in Fleet -> Settings -> Outputs -> Advanced YAML configuration.

  bulk_max_size: 4096
  worker: 1
  queue.mem.events: 8192
  queue.mem.flush.min_events: 4096
  queue.mem.flush.timeout: 5s
  compression_level: 5
  connection_idle_timeout: 3s

When I check the index stats, both size_in_bytes and total_data_set_size_in_bytes are the same, indicating that the compression has not been successful. Can anyone suggest what might be wrong with my configuration? Thank you.

  "store": {
    "size_in_bytes": 304600,
    "total_data_set_size_in_bytes": 304600,
    "reserved_in_bytes": 0
  },

[leandrojmp (Leandro Pereira)]

Those are different things.

The compression configuration in the Elastic Agent that you mentioned is related to the compression of the request that will be sent to Elasticsearch, it has no influence in the compression of the data in Elasticsearch itself after it is indexed.

The compressions of the data in Elasticsearch is managed by a setting on the index template named index.codec, there are 2 options default and best_compression.

Since you are using Elastic Agent and its integrations, the compression is already set to best_compression.

[wangsubo]

Thank you for your response. I have checked the index template and confirmed that it is set to best_compression. However, the values for both size_in_bytes and total_data_set_size_in_bytes remain the same.

How can I determine whether the compression has been successful or not?

[leandrojmp (Leandro Pereira)]

Not sure what you want to determine, if the index codec is set the best_compression then the stored data is already compressed.

If I'm not wrong the value you see in size_in_bytes is after the compression, is the value used by the underlying data in disk.