Metricbeat cannot use fields in template name

[errupeshmca (Heyan)]

Elasticsearch version 8.5.2

metricbeat version 8.5.2

I am getting this issue when trying to customize template pattern in metricbeat below is my configuration, I want to use data stream and but not the default one, the customize one, so that I can separate data stream for per client wise.

Please let me know if that possible or what I am doing wrong here.

{"log.level":"error","@timestamp":"2022-12-09T17:14:41.781Z","log.origin":{"file.name":"instance/beat.go","file.line":1057},"message":"Exiting: error loading template: error creating template instance: key not found","service.name":"metricbeat","ecs.version":"1.6.0"}
Exiting: error loading template: error creating template instance: key not found

metricbeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false

setup.template.settings:
  index.number_of_shards: 1
  index.codec: best_compression

setup.kibana.host: "http://master.example.com:5601"
setup.kibana.username: "elastic"
setup.kibana.password: "xxxxx"


output.elasticsearch:
  index: "metricbeat-%{[agent.version]}.%{[data_stream.namespace]}.%{[data_stream.dataset]}.%{[data_stream.type]}"
  hosts: ["https://master.example.com:9200"]
  username: "elastic"
  password: "xxxxx"
  ssl:
    certificate_authorities: ['/etc/metricbeat/config/certs/http_ca.crt']

processors:
  - add_host_metadata: ~
  - add_cloud_metadata: ~
  - add_docker_metadata: ~
  - add_kubernetes_metadata: ~

fields:
  service:
    environment: "prod"
    type: "system"
    name: "http"

tags: ["prod"]
data_stream.dataset: logs
data_stream.namespace: prod
data_stream.type: http

setup.ilm.overwrite: true
setup.template.enabled: true
#setup.template.name: "metricbeat-%{[agent.version]}.%{[service.name]}"
setup.template.name: "metricbeat-%{[agent.version]}.%{[data_stream.namespace]}.%{[data_stream.dataset]}.%{[data_stream.type]}"
setup.template.overwrite: false
setup.template.pattern: "metricbeat-%{[agent.version]}.%{[data_stream.namespace]}.%{[data_stream.dataset]}.%{[data_stream.type]}"
setup.dashboards.index: "metricbeat-*"

[stephenb (Stephen Brown)]

@errupeshmca

data_stream.dataset: logs
data_stream.namespace: prod
data_stream.type: http

Those fields are only available in events... They could be used in the output section but not in the template section.

Pretty sure These fields are not available during setup so you're trying to use them in a template name which is constructed during setup phase. Pretty sure that's what the error is.

They're a limited number of fields that are available during the setup phase.

You would need to hardcode those values into the template name... Or just keep the template name generic and then use them in the output section.