Metricbeat and Heartbeat 8.x custom index Name should pickup from template ,but creates always bedefault index name

Elastic Stack Beats
13

Did you simply try to replace the generated ILM Policy and Template with yours with the same names etc after you ran setup?

Data streams rollover that is part of the core functionality... but you are right it is a bit different in 8.x Data Streams it is internal you do not use the write alias the Data Stream it IS the alias.

So what I did to test is I ran the setup with the config I had above.

Then you can just edit the ILM policy to what you want it to be
Edit the template to what you want it to be but there are a few key differences see below...

{
  "index_templates": [
    {
      "name": "cis-metric-8.8.0",
      "index_template": {
        "index_patterns": [
          "cis-metric-8.8.0" <!-- NOTE IMPORTANT The name of the Data Stream
        ],
        "template": {
          "settings": {
            "index": {
              "lifecycle": {
                "name": "cis-metric"
              },
              "codec": "best_compression",
              "mapping": {
                "total_fields": {
                  "limit": "10000"
                }
              },
              "refresh_interval": "5s",
              "number_of_shards": "1",
              "max_docvalue_fields_search": "200",
              "query": {
                "default_field": [
                  "message",
                  "tags"
                ]
              }
            }
          },
          "mappings": {
            "_meta": {
              "beat": "metricbeat",
              "version": "8.8.0"
            },
            "dynamic_templates": [
              ......
              ],
            "date_detection": false,
            "properties": {
              ....... Mappings Here
            }
          }
        },
        "composed_of": [],
        "priority": 150,
        "data_stream": {  <!--- NOTE IMPORTANT SUPER IMPORTANT I THINK YOU ARE MISSING THIS
          "hidden": false,
          "allow_custom_routing": false
        }
      }
    }
  ]
}

Then my metricbeat.yml looks like this...

setup.ilm.enabled: false
setup.template.enabled: false
output.elasticsearch:
  hosts: ["https://localhost:9200"]
  index: "cis-metric-%{[agent.version]}"   
  username: "elastic"
  password: "password"
  ssl.verification_mode: "none"

And now when I run metricbeat the data goes the correct places / data stream.

Screenshot 2023-07-08 at 10.34.37 AM
Screenshot 2023-07-08 at 10.34.37 AM2446×1296 255 KB

YOU can do all this manually as well setup you your entire data stream by following this

I like the setup method because it creates everything for me then I just edit and change to what I want

NOTE you are right you can run the setup with just this and it set up the ILM, Data Stream and Template the other stuff not needed any more

setup config

setup.ilm.enabled: true
setup.ilm.check_exists: true
setup.ilm.policy_name: cis-metric


setup.template.enabled: true  # <!--- You should explicitly set this 
setup.template.name: "cis-metric-%{[agent.version]}" # <!----This needs to match the index name because now it is a data stream
setup.template.pattern: "cis-metric-%{[agent.version]}" # <!-- Better matching Hygiene because it is a data stream 
setup.template.overwrite: false # <!--- Careful with this if set to true it will overwrite every time.

So run setup like that.

Then edit the ILM policy to what you want
Edit the Template Mappings to what you want and you are good...

If you save all that stuff you can skip the setup... just create them and then run with your custom data stream name.

run config

setup.ilm.enabled: false
setup.template.enabled: false
output.elasticsearch:
  hosts: ["https://localhost:9200"]
  index: "cis-metric-%{[agent.version]}"   
  username: "elastic"
  password: "password"
  ssl.verification_mode: "none"

hope this helps