Metricbeat does not send documents to indices other than .monitoring-es-*-mb, even when output.elasticsearch.index specified

[Olivia_Anderson]

I have 2 elasticsearch clusters deployed with ECK, and want to send stack monitoring data from the main cluster to the monitoring cluster. Metricbeat is setup with the following configs:

elasticsearch.yml:

- module: elasticsearch
  enabled: true
  metricsets:
    - ccr
    - cluster_stats
    - enrich
    - index
    - index_recovery
    - index_summary
    - ml_job
    - node_stats
    - shard
  period: 10s
  hosts: ["${MONITORED_ES_HOST}:${MONITORED_ES_PORT}"]
  username: "${MONITORED_ES_USERNAME}"
  password: "${MONITORED_ES_PASSWORD}"
  ssl.verification_mode: "none"
  xpack.enabled: true

metricbeat.yml

metricbeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false

processors:
  - add_cloud_metadata:

output.elasticsearch:
  hosts: ['${MONITORING_ES_HOST}:${MONITORING_ES_PORT}']
  username: ${MONITORING_ES_USERNAME}
  password: ${MONITORING_ES_PASSWORD}
  ssl.verification_mode: "none"

output.elasticsearch.index: "metricbeat-es-%%{[agent.version]}-%%{+yyyy.MM.dd}"
setup.template.name: "metricbeat-es"
setup.template.pattern: "metricbeat-es-*"
setup.ilm.enabled: false
logging.level: debug

The index template is loaded correctly, under the correct name, though data only flows into .monitoring-es-7-mb* indices.

I've tried multiple versions of metricbeat (from 7.7.0-latest), and verified that the configuration is loading correctly with metricbeat test config -e, including output.elasticsearch.index. I've also tried using ECK to deploy metricbeat with autodiscover, and using a generic kubernetes deployment with no luck. The only time data was placed into the expected index was when I had mistyped the hostname of the main cluster and error logs were stored.

Is there a way to change the output index when using the elasticsearch module only?

[matschaffer]

This is the key configuration here.

@Olivia_Anderson

xpack.enabled: true

That tells metricbeat to send data able to drive the "Stack Monitoring" UI in kibana. If you take that out you should be able to override the output index, but you won't be able to use the Stack Monitoring kibana application, you'll need to build your own visualisations.