_source filed is using to much disk space

_source for metricbeat index is using a lot of space. Index size is 15.2gb and _source filed is 12.2gb. And from 4 kubernetes nodes in one day total size of metricbeat size is 160 gb. Any ideas why metricbeat index size so big and how to reduce it?

Unfortunately, _source is just one of those things that currently just takes up space. You didn't mention which version of Elasticsearch/Metricbeat you're currently using, but you can take a look at: Tune for disk usage | Elasticsearch Guide [8.3] | Elastic to see if there are any changes you could make to improve this.

Two things to note, there are 2 features being worked on on the Elasticsearch side that I think will greatly improve storage efficiency in the future:

  1. TSDB - Add better support for metric data types (TSDB) · Issue #74660 · elastic/elasticsearch · GitHub this from what I can tell will allow for better optimization of storing metrics (like data from Metricbeat)
  2. Synthetic Source - Synthetic Source · Issue #86603 · elastic/elasticsearch · GitHub this seems like it would effectively remove the overhead of _source while not having all the drawbacks of fully disabling _source.

Note: Both of the above changes I don't think are GA in any current release of Elasticsearch.

1 Like

I am using version 8.3.2. Thank you for link. I will take a look