Elasticsearch monitoring by metricbeat creating index with pattern .ds-.monitoring-es-8-mb-yyyy.mm.dd-*

[ramdas]

Hi,

I have recently started testing to use metricbeat for elasticsearch's monitoring. i have elasticsearch deployed as statefulset and metricbeat as daemonset on azure kubernetes services cluster.

i have configured metricbeat module as below:

- module: elasticsearch
          xpack.enabled: true
          scope: node
          metricsets:
            - node
            - node_stats
            - index
            - index_recovery
            - index_summary
            - shard
            - cluster_stats
          period: 10s
          hosts: [https://localhost:${ES_NODEPORT}"]
          username: "${ELASTICSEARCH_USERNAME}"
          password: "${ELASTICSEARCH_PASSWORD}"
          ssl.verification_mode: "certificate"

also i have configured output.elasticsearch as follows:

output.elasticsearch:
          hosts: "[${ELASTICSEARCH_HOST}:9200]"
          protocol: "${ELASTICSEARCH_PROTOCOL:https}"
          username: "${ELASTICSEARCH_USERNAME:elastic}"
          ssl.verification_mode: "${ELASTICSEARCH_VERIFYSSL:none}"
          password: "${ELASTICSEARCH_PASSWORD:welcome1}"
          index: "${INDEX_NAME}-%{[agent.version]}-%{+yyyy.MM.dd}"
          allow_older_versions: true

same time i am using metricbeat to monitor kubernetes cluster metrics too. after everything is deployed metricbeat scrapes metrics for kubernetes cluster and indexes them into elasticsearch with index pattern like "${INDEX_NAME}-%{[agent.version]}-%{+yyyy.MM.dd}" as given in output.elasticsearch which is as expected. however the elasticsearch monitoring data is indexed into different index with pattern like ".ds-.monitoring-es-8-mb-yyyy.mm.dd-* " which is backed by datastream ".monitoring-es-8-mb" which is unexpected.

point to note here is i am not using ILM here and controlling index creation/rollover using index pattern in metribeat. the new indexes created for kubernetes metrics are created midnight 00:00 as per index pattern, however new indexes created for elasticsearch's monitoring are not created at midnight, and they get created around the time when i first configured metricbeat/first elasticsearch monitoring index was created.

now i am setting up dashboard for elasticsearch's monitoring in grafana with data i have in elasticsearch and i cant find the index for suppose today 2023.02.21 because there is no index created with 2023.02.21 date pattern in index name as new index would be created later in evening.

is there any solution with which i can either merge two different indexes being created by same metricbeat deployment or configure elasticsearch monitoring indexes to bb rollover at midnight without ILM.