Limit es monitored data in metricbeat

Elastic Observability Metrics
1

Hello,

I’m currently monitoring ELK stack through metricbeat (Elasticsearch, Kibana & Logstash)
but the monitoring index for Elasticsearch “.ds-.monitoring-es…….” is around 15GB per day so I switched the period to 60 seconds instead of 10, but I think the size of the index is way to big compared to the data provided.

So my question is, how do I limit the info monitored by Metricbeat without losing essential data?
I tried to inspect the index and found out that most of the fields are empty.

image
image1488×234 21.5 KB

2

Hello @SamehSaeed

Could you please share the number of nodes in your cluster for which this monitoring is enabled along with 1 of the dummy <metricbeat.yml > as to see what all modules as enabled.

Also share the output for below query via devtools :

GET _cat/indices/.ds-.monitoring-es*?v&s=store.size:desc

Thanks!!

3

Hello, thank you for your reply

I have only one node

metricbeat.yml :

metricbeat.config.modules:
  # Glob pattern for configuration loading
  path: ${path.config}/modules.d/*.yml

  # Set to true to enable config reloading
  reload.enabled: true

  # Period on which files under path should be checked for changes
  reload.period: 60s

# ======================= Elasticsearch template setting =======================

setup.template.settings:
  index.number_of_shards: 1
  index.codec: best_compression
  #_source.enabled: false

name: "****"

logging.level: error

processors:
  - add_host_metadata: ~
  - add_cloud_metadata: ~
  - add_docker_metadata: ~
  - add_kubernetes_metadata: ~

output.elasticsearch:

  hosts: ["https://****:****"]
  protocol: "https"
  username: "remote_monitoring_user"
  password: "****"
  ssl.certificate_authorities: ["****"]
  ssl.verification_mode: certificate

  preset: balanced
  
  setup.kibana:
  host: "https://****:****"

elasticsearch-xpack.yml :

# Module: elasticsearch
# Docs: https://www.elastic.co/guide/en/beats/metricbeat/8.x/metricbeat-module-elasticsearch.html

- module: elasticsearch
  xpack.enabled: true
  period: 60s
  hosts: ["https://****:****"]
  username: "elastic"
  password: "*****"
  ssl.verification_mode: "none"
  scope: node
  metricsets:
    - node
#api_key: "foo:bar"

GET _cat/indices/.ds-.monitoring-es*?v&s=store.size:desc :

health status index                                     uuid                   pri rep docs.count docs.deleted store.size pri.store.size dataset.size
green  open   .ds-.monitoring-es-8-mb-2025.08.19-000197 dO4fIXr8TsGBlygeNEN4fg   1   0   24154649            0     14.9gb         14.9gb       14.9gb
green  open   .ds-.monitoring-es-8-mb-2025.08.20-000199 aByXutjDTFO5OuykEsSt7g   1   0    5169566            0      3.1gb          3.1gb        3.1gb

.ds-.monitoring-es-8-mb-2025.08.20-000199 is much smaller because the period is set to 60s