Metricbeat and Heartbeat 8.x custom index Name should pickup from template ,but creates always bedefault index name

[PRASHANT_MEHTA]

Hello All,

I'm migrating metricbeat and heartbeat from 7.9.1 to 8.7.1 version and current challenge is I'm unbale to create custom index name defined in my index template and metricbeat.yml.
I'm not sure how come everytime a default index by metricbeat is getting created,requirement is to have my own index name.

This is the output not required(8.7.1):

The index that should be shown up is,this was achieved in 7.9.1 using below configs and need same in 8.7.1:
cis-metric-2023.07.04-000001

metricbeat.yml:

metricbeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false
setup.ilm.enabled: true
setup.ilm.check_exists: true
setup.ilm.rollover_alias: cis-metric
setup.ilm.pattern: '{now/d}-000001'
setup.ilm.overwrite: false
setup.template.settings:
  index.number_of_shards: 1
setup.kibana:
  host: https://abc:5601
output.elasticsearch:
  hosts: ["abc:443"]
  ssl.certificate_authorities: ["/l/ppp/mis/config/ece_proxy_root_ca.pem"]
  protocol: "https"
  username: "b"
  password: "b"
processors:
- rename:
    fields:
    - from: agent.hostname
      to: host.name
    ignore_missing: true
    fail_on_error: false
- drop_fields:
    fields:
    - host.mac
    - ecs
    - agent.ephemeral_id
    - agent.type
    - agent.version
    - agent.hostname
    - event
monitoring.enabled: true
monitoring.elasticsearch: null

template:

PUT _index_template/cis-metric
{
  "template": {
    "settings": {
      "index": {
        "lifecycle": {
          "name": "cis-monitoring-common-policy",
          "rollover_alias": "cis-metric"
        },
        "number_of_shards": "1",
        "number_of_replicas": "0"
      }
    },
    "mappings": {
      "_routing": {
        "required": false
      },
      "numeric_detection": false,
      "dynamic_date_formats": [
        "strict_date_optional_time",
        "yyyy/MM/dd HH:mm:ss Z||yyyy/MM/dd Z"
      ],
      "_source": {
        "excludes": [],
        "includes": [],
        "enabled": true
      },
      "dynamic": true,
      "date_detection": true,
      "properties": {
        "system.cpu.iowait.norm.pct": {
          "scaling_factor": 1000,
          "type": "scaled_float"
        },
        "system.process.memory.rss.pct": {
          "scaling_factor": 1000,
          "type": "scaled_float"
        },
        "system.diskio.write.bytes": {
          "type": "long"
        },
        "system.load.cores": {
          "type": "long"
        },
        "system.process.cpu.total.value": {
          "type": "long"
        },
        "service.type": {
          "type": "keyword"
        },
        "system.diskio.iostat.service_time": {
          "type": "float"
        }
      }
    }
  },
  "index_patterns": [
    "cis-metric-*"
  ],
  "composed_of": []
}

Any suggestion would be helpful.

[system]

7.9.1 is EOL and no longer supported. Please upgrade ASAP.

(This is an automated response from your friendly Elastic bot. Please report this post if you have any suggestions or concerns )

[stephenb (Stephen Brown)]

Hi @PRASHANT_MEHTA Not sure why the above was working for 7.9 ... seems like you never named the index in the output section so I would expect the index to be the default name.

Anyways...

Here is a working version for 8.x/8.7/8.8 etc

This sets up a template, data stream, index lifecycle policy, and backing indices etc

run setup first

You could even setup the dashboards to work with this if you want...

output.elasticsearch:
  hosts: ["https://localhost:9200"]
  index: "cis-metric-%{[agent.version]}"   # <!---- Don't add -%{+yyyy.MM.dd} its a data strean and getts added to the backing index automatically
  username: "elastic"
  password: "password"
  ssl.verification_mode: "none"

setup.ilm.enabled: true
setup.ilm.check_exists: true
setup.ilm.rollover_alias: cis-metric
setup.ilm.pattern: '{now/d}-000001'
setup.ilm.overwrite: false

setup.template.enabled: true  # <!--- You should explicitly set this 
setup.template.name: "cis-metric-%{[agent.version]}" # <!----This needs to match the index name because now it is a data stream
setup.template.pattern: "cis-metric-%{[agent.version]}" # <!-- Better matching Hygiene because it is a data stream 
setup.template.overwrite: false # <!--- Careful with this if set to true it will overwrite every time.
setup.ilm.policy_name: cis-metric
# setup.dashboards.enabled: true # Uncomment this during setup if you want to load the dashboard to work with this data stream. 

[PRASHANT_MEHTA]

Hello @stephenb ,

Thanx for your time to look into this and assisting on this.
I did same setting as yours but still the data is not coming in index.
If you will see above my custom template name is: PUT _index_template/cis-metric, do I need to do this PUT _index_template/cis-metric-8.7.1(this also didn't help)

My index is getting created like yours but data(docs) is not coming. I m still also getting by default metricbeat index getting created.
I'd require metricbeat data to be mapped with my own custom template shared above.Not sure what is wrong .

Index templates:

Everytime metricbeats starts this index also comes up:

Below is my current metricbeat.yml and you can have look at template above: template name and index pattern extreme end,does template needs any change so that it metricbeat.yml understands to map my external template?

Can you also plz tell me if I dont want to use data stream and manage it in old way,how this can be done- cis-metric-date-000001.

metricbeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false
setup.ilm.enabled: true
setup.ilm.check_exists: true
setup.ilm.rollover_alias: cis-metric
setup.ilm.pattern: '{now/d}-000001'
setup.ilm.overwrite: false
setup.template.enabled: true
setup.template.name: "cis-metric-%{[agent.version]}"
setup.template.pattern: "cis-metric-%{[agent.version]}"
setup.template.overwrite: false
setup.ilm.policy_name: cis-metric-policy
setup.template.settings:
  index.number_of_shards: 1
setup.kibana:
  host: https://abc:5601
output.elasticsearch:
  hosts: ["https://abc:443"]
  index: "cis-metric-%{[agent.version]}"
  ssl.certificate_authorities: ["/l/ppp/cis/config/ece_atr_root_ca.pem"]
  protocol: "abc"
  username: "abc"
  password: "abc"
processors:
- rename:
    fields:
    - from: agent.hostname
      to: host.name
    ignore_missing: true
    fail_on_error: false
- drop_fields:
    fields:
    - host.mac
    - ecs
    - agent.ephemeral_id
    - agent.type
    - agent.version
    - agent.hostname
    - event
monitoring.enabled: true
monitoring.elasticsearch: null

[stephenb (Stephen Brown)]

EDIT

start in the foreground with this and look at the docs that are being published

filebeat -e -d "*"

Also, are there any errors in the logs?

[stephenb (Stephen Brown)]

@PRASHANT_MEHTA

monitoring.enabled: true
monitoring.elasticsearch: null

That is why you have the .monitoring... Index

Set to false to turn it off or take out both those lines

I don't think you're writing any data at all.

Do you have the modules.d/system.yml enabled

It is not clear to me if the 2 templates are colliding...

[PRASHANT_MEHTA]

Hello,

The data is available and now I can see in index,I tried with your method and still didn't got the data in index.I suppose the rollover thing is removed in 8.X version reading docs.
Now I wrote new metricbeat.yml and my index is getting created and ilm is also applied.But after every 10 to 15 mins I'm getting this error:

system.yml was enabled and getting data now.

Updated metricbeat.yml

metricbeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false
setup.ilm.enabled: true
setup.ilm.check_exists: true
# setup.ilm.rollover_alias: cis-metric
# setup.ilm.pattern: '{now/d}-000001'
setup.ilm.overwrite: false
setup.template.enabled: true
setup.template.name: "cis-metric"
**setup.template.pattern: "cis-metric-*"**
setup.template.overwrite: false
setup.ilm.policy_name: cis-metric-policy
setup.template.settings:
  index.number_of_shards: 1
setup.kibana:
  host: https://abc:5601
output.elasticsearch:
  hosts: ["https://abc:443"]
  **index: "cis-metric-%{+yyyy.MM.dd}"**
  ssl.certificate_authorities: ["/l/app/ece_proxy_root_ca.pem"]
  protocol: ""
  username: ""
  password: ""
processors:
- rename:
    fields:
    - from: agent.hostname
      to: host.name
    ignore_missing: true
    fail_on_error: false
- drop_fields:
    fields:
    - host.mac
    - ecs
    - agent.ephemeral_id
    - agent.type
    - agent.version
    - agent.hostname
    - event

In template if you will check earlier I shared,have removed rollover from it, as in 8.x its not supported I guess.

Removed this also:
monitoring.enabled: true
monitoring.elasticsearch: null

[PRASHANT_MEHTA]

Again refreshed error is gone.Whats happening that sometimes it occurs

[stephenb (Stephen Brown)]

can you please try exactly my example

@PRASHANT_MEHTA

**index: "cis-metric-%{+yyyy.MM.dd}"**

Not correct that is what is causing the issue..

Can you please try Exactly my config just change the host and u/p.

output.elasticsearch:
  hosts: ["https://localhost:9200"]
  index: "cis-metric-%{[agent.version]}" 
  username: "elastic"
  password: "password"
  ssl.verification_mode: "none"

setup.ilm.enabled: true
setup.ilm.check_exists: true
setup.ilm.rollover_alias: cis-metric
setup.ilm.pattern: '{now/d}-000001'
setup.ilm.overwrite: false

setup.template.enabled: true  
setup.template.name: "cis-metric-%{[agent.version]}" 
setup.template.pattern: "cis-metric-%{[agent.version]}" 
setup.template.overwrite: false 
setup.ilm.policy_name: cis-metric

[PRASHANT_MEHTA]

Applied the same config you provided.Below is how it looks like,but the docs in index are not coming. May I know if you are using your own custom template defined for mapping fields coming from metricbeat?

TWO TEMPLATES:
cis-metric : I've committed in devtools-has my own mappings.
cis-metric-8.7.1 creates automatically ,from metricbeat.yml

How demo templates looks like:

PUT _index_template/cis-metric
{
  "template": {
    "settings": {
      "index": {
        "lifecycle": {
          "name": "cis-metric-policy",
          "rollover_alias": "cis-metric"
        },
        "number_of_shards": "1",
        "number_of_replicas": "0"
      }
    },
    "mappings": {
      "_routing": {
        "required": false
      },
      "numeric_detection": false,
      "dynamic_date_formats": [
        "strict_date_optional_time",
        "yyyy/MM/dd HH:mm:ss Z||yyyy/MM/dd Z"
      ],
      "_source": {
        "excludes": [],
        "includes": [],
        "enabled": true
      },
      "dynamic": true,
      "date_detection": true,
      "properties": {
        "system.cpu.iowait.norm.pct": {
          "scaling_factor": 1000,
          "type": "scaled_float"
        }
      }
    }
  },
  "index_patterns": [
    "cis-metric-*"
  ],
  "composed_of": []
}

[stephenb (Stephen Brown)]

No I did not create a custom template metricbeat does it automatically... I think they are colliding or something...

Can you take your custom template out and try... clean up everything and try again...
then we can work backwards to your custom template.

and I can see it is not the same because the policy name is wrong

Mine is this..
setup.ilm.policy_name: cis-metric

Your shows : cis-metric-policy

Which means your template is being used... not the automatically generated template referenced with my configuration.

I suspect there is some mapping issue in your custom template vs the auto generated one...

Did you actually look at the logs or run in the foreground with the debug flags I showed you?

[PRASHANT_MEHTA]

Hello @stephenb ,

My bad,I did mistake in metricbeat.yml, your above config works fine and now I'm getting the data in my index.
Mistake done was added this: index: "cis-metric-%{+yyyy.MM.dd}",removed and worked fine with your congigs now.Thanx for you efforts on this.

Few doubts if you could help me with this:

1)I was using 7.9.1 earlier and now migrating to 8.7.1 version. I'm facing challenges ,everything worked fine in 7.9.1 and now in 8.7.1 data stream came.I'm exploring and reading documentation but unable to implement or get the data in beats indices like I want.

I've around 50 indices i.e these are such that logstash process and send data to them and these documents are frequently updated. Through documentation understood DATA STREAM are not recommended for such data and only suitable only for time series data.Correct me if I am wrong.

2)Real challenge now: I want to use metricbeat,filebeat and heartbeat(all time series data) now with my own CUSTOM templates following my own ILM policy defined.
I did this thing in 7.9.1 and worked everything well. Now in 8.7.1 what is the method and how to implement I'm not getting exactly after reading documentation.

Rollover are also not supported in 8.7.1 I guess? , and I use it in my template earlier.

Let me know how this can be done. Plz glance my Metric beat custom template and how data from beats could be mapped to custom templates?
Ofcourse I tried everything from documentation ,with no luck of data stream getting created for custom template/policy,and not sure now to achieve this.Its work in progress for now.

CRUX: I would request to help out how to send data to respective indices following custom template and custom policy. Not using by default ones.
For now with data streams seems confusing, rollover not supported ,then where could i define rollover alias type setting.
I guess without using data streams also this could be done,but did'nt got relevant docs.

I dont think below are valid for 8.7.1

setup.ilm.rollover_alias: cis-metric
setup.ilm.pattern: '{now/d}-000001'

Metricbeat Template:(removed rollover_alias from settings in template,7.9.1 supported,8.7.1 dont support.

PUT _index_template/cis-metric
{
  "template": {
    "settings": {
      "index": {
        "lifecycle": {
          "name": "cis-metricbeat-policy"
        },
        "number_of_shards": "1",
        "number_of_replicas": "0"
      }
    },
    "mappings": {
      "_source": {
        "excludes": [],
        "includes": [],
        "enabled": true
      },
      "_routing": {
        "required": false
      },
      "dynamic": true,
      "numeric_detection": false,
      "date_detection": true,
      "dynamic_date_formats": [
        "strict_date_optional_time",
        "yyyy/MM/dd HH:mm:ss Z||yyyy/MM/dd Z"
      ],
      "properties": {
        "@timestamp": {
          "type": "date"
        },
        "agent.id": {
          "type": "keyword"
        },
        "agent.name": {
          "type": "keyword"
        },
        "cpu.status": {
          "type": "keyword"
        },
        "host.name": {
          "type": "text",
          "fields": {
            "keyword": {
              "ignore_above": 256,
              "type": "keyword"
            }
          }
        },
        "metricset.name": {
          "type": "keyword"
        },
        "metricset.period": {
          "type": "long"
        },
        "mountpoint.details": {
          "type": "keyword"
        },
        "process.args": {
          "type": "keyword"
        },
        "process.details": {
          "type": "keyword"
        },
        "process.name": {
          "type": "text",
          "fields": {
            "keyword": {
              "ignore_above": 256,
              "type": "keyword"
            }
          }
        },
        "process.pgid": {
          "type": "long"
        },
        "process.pid": {
          "type": "long"
        },
        "process.ppid": {
          "type": "long"
        },
        "process.type": {
          "type": "keyword"
        },
        "service.type": {
          "type": "keyword"
        },
        "system.cpu.cores": {
          "type": "long"
        },
        "system.cpu.idle.norm.pct": {
          "scaling_factor": 1000,
          "type": "scaled_float"
        },
        "system.cpu.idle.pct": {
          "scaling_factor": 1000,
          "type": "scaled_float"
        },
        "system.cpu.iowait.norm.pct": {
          "scaling_factor": 1000,
          "type": "scaled_float"
        },
        "system.cpu.iowait.pct": {
          "scaling_factor": 1000,
          "type": "scaled_float"
        },
        "system.cpu.irq.norm.pct": {
          "scaling_factor": 1000,
          "type": "scaled_float"
        },
        "system.cpu.irq.pct": {
          "scaling_factor": 1000,
          "type": "scaled_float"
        },
        "system.cpu.nice.norm.pct": {
          "scaling_factor": 1000,
          "type": "scaled_float"
        },
        "system.cpu.nice.pct": {
          "scaling_factor": 1000,
          "type": "scaled_float"
        },
        "system.cpu.softirq.norm.pct": {
          "scaling_factor": 1000,
          "type": "scaled_float"
        },
        "system.cpu.softirq.pct": {
          "scaling_factor": 1000,
          "type": "scaled_float"
        },
        "system.cpu.steal.norm.pct": {
          "scaling_factor": 1000,
          "type": "scaled_float"
        },
        "system.cpu.steal.pct": {
          "scaling_factor": 1000,
          "type": "scaled_float"
        },
        "system.cpu.system.norm.pct": {
          "scaling_factor": 1000,
          "type": "scaled_float"
        },
        "system.cpu.system.pct": {
          "scaling_factor": 1000,
          "type": "scaled_float"
        },
        "system.cpu.total.norm.pct": {
          "scaling_factor": 1000,
          "type": "scaled_float"
        },
        "system.cpu.total.pct": {
          "scaling_factor": 1000,
          "type": "scaled_float"
        },
        "system.cpu.user.norm.pct": {
          "scaling_factor": 1000,
          "type": "scaled_float"
        },
        "system.cpu.user.pct": {
          "scaling_factor": 1000,
          "type": "scaled_float"
        },
        "system.diskio.io.ops": {
          "type": "long"
        },
        "system.diskio.io.time": {
          "type": "long"
        },
        "system.diskio.iostat.await": {
          "type": "float"
        },
        "system.diskio.iostat.busy": {
          "type": "float"
        },
        "system.diskio.iostat.queue.avg_size": {
          "type": "float"
        },
        "system.diskio.iostat.read.await": {
          "type": "float"
        },
        "system.diskio.iostat.read.per_sec.bytes": {
          "type": "float"
        },
        "system.diskio.iostat.read.request.merges_per_sec": {
          "type": "float"
        },
        "system.diskio.iostat.read.request.per_sec": {
          "type": "float"
        },
        "system.diskio.iostat.request.avg_size": {
          "type": "float"
        },
        "system.diskio.iostat.service_time": {
          "type": "float"
        },
        "system.diskio.iostat.write.await": {
          "type": "float"
        },
        "system.diskio.iostat.write.per_sec.bytes": {
          "type": "float"
        },
        "system.diskio.iostat.write.request.merges_per_sec": {
          "type": "float"
        },
        "system.diskio.iostat.write.request.per_sec": {
          "type": "float"
        },
        "system.diskio.name": {
          "type": "keyword"
        },
        "system.diskio.read.bytes": {
          "type": "long"
        },
        "system.diskio.read.count": {
          "type": "long"
        },
        "system.diskio.read.time": {
          "type": "long"
        },
        "system.diskio.serial_number": {
          "type": "keyword"
        },
        "system.diskio.write.bytes": {
          "type": "long"
        },
        "system.diskio.write.count": {
          "type": "long"
        },
        "system.diskio.write.time": {
          "type": "long"
        },
        "system.diskspace.usage": {
          "type": "long"
        },
        "system.filesystem.used.pct": {
          "type": "float"
        },
        "system.fsstat.count": {
          "type": "long"
        },
        "system.fsstat.status": {
          "type": "keyword"
        },
        "system.fsstat.total_files": {
          "type": "long"
        },
        "system.fsstat.total_size.free": {
          "type": "long"
        },
        "system.fsstat.total_size.total": {
          "type": "long"
        },
        "system.fsstat.total_size.used": {
          "type": "long"
        },
        "system.load.1": {
          "scaling_factor": 100,
          "type": "scaled_float"
        },
        "system.load.15": {
          "scaling_factor": 100,
          "type": "scaled_float"
        },
        "system.load.5": {
          "scaling_factor": 100,
          "type": "scaled_float"
        },
        "system.load.cores": {
          "type": "long"
        },
        "system.load.norm.1": {
          "scaling_factor": 100,
          "type": "scaled_float"
        },
        "system.load.norm.15": {
          "scaling_factor": 100,
          "type": "scaled_float"
        },
        "system.load.norm.5": {
          "scaling_factor": 100,
          "type": "scaled_float"
        },
        "system.memory.actual.free": {
          "type": "long"
        },
        "system.memory.actual.used.bytes": {
          "type": "long"
        },
        "system.memory.actual.used.pct": {
          "scaling_factor": 100,
          "type": "scaled_float"
        },
        "system.memory.free": {
          "type": "long"
        },
        "system.memory.swap.free": {
          "type": "long"
        },
        "system.memory.swap.total": {
          "type": "long"
        },
        "system.memory.swap.used.bytes": {
          "type": "long"
        },
        "system.memory.swap.used.pct": {
          "scaling_factor": 1000,
          "type": "scaled_float"
        },
        "system.memory.total": {
          "type": "long"
        },
        "system.memory.used.bytes": {
          "type": "long"
        },
        "system.memory.used.pct": {
          "scaling_factor": 1000,
          "type": "scaled_float"
        },
        "system.network.in.bytes": {
          "type": "long"
        },
        "system.network.in.dropped": {
          "type": "long"
        },
        "system.network.in.errors": {
          "type": "long"
        },
        "system.network.in.packets": {
          "type": "long"
        },
        "system.network.name": {
          "type": "keyword"
        },
        "system.network.out.bytes": {
          "type": "long"
        },
        "system.network.out.dropped": {
          "type": "long"
        },
        "system.network.out.errors": {
          "type": "long"
        },
        "system.network.out.packets": {
          "type": "long"
        },
        "system.process.cpu.start_time": {
          "type": "date"
        },
        "system.process.cpu.total.norm.pct": {
          "scaling_factor": 1000,
          "type": "scaled_float"
        },
        "system.process.cpu.total.pct": {
          "scaling_factor": 1000,
          "type": "scaled_float"
        },
        "system.process.cpu.total.value": {
          "type": "long"
        },
        "system.process.memory.rss.bytes": {
          "type": "long"
        },
        "system.process.memory.rss.pct": {
          "scaling_factor": 1000,
          "type": "scaled_float"
        },
        "system.process.memory.share": {
          "type": "long"
        },
        "system.process.memory.size": {
          "type": "long"
        },
        "system.process.state": {
          "type": "keyword"
        },
        "user.name": {
          "type": "text",
          "fields": {
            "keyword": {
              "ignore_above": 256,
              "type": "keyword"
            }
          }
        },
        "system.process.cpu.total.ticks":
        {
          "type": "long"
        }
    }}
  },
  "index_patterns": [
    "cis-metric-*"
  ],
  "composed_of": []
}

system.yml(modules.d)

# Module: system
# Docs: https://www.elastic.co/guide/en/beats/metricbeat/7.8/metricbeat-module-system.html
- module: system
  period: 30m
  metricsets:
    - memory
 
- module: system
  period: 10s
  metricsets:
    - load
    - network

- module: system
  period: 30m
  metricsets:
    - fsstat
    - filesystem
  filesystem.ignore_types: [ssysfs, rootfs, ramfs, bdev, cgroup, cpuset,debugfs, securityfs, sockfs, dax, bpf, pipefs, anon_inodefs, configfs, devpts, hugetlbfs, autofs, pstore, mqueue, selinuxfs, rpc_pipefs, binfmt_misc, overlay,nfs4,nfs,proc,sysfs]
  processors:
  - script:
        lang: javascript
        id: MountPointDataAdapter
        file: MountPointDataAdapter.js
        params:
            hostname: '${HOSTNAME}'
        
- module: system
  period: 10s
  metricsets:
    - diskio
  diskio.include_devices: ["sda", "sda1","vda","vda1"]
    
- module: system
  period: 5m
  metricsets:
    - process
  processes: ['.*']
  process.include_cpu_ticks: true
  processors:
  - script:
        lang: javascript
        id: ProcessDataUpdater
        file: ProcessDataUpdater.js
        params:
            hostname: '${HOSTNAME}'
  - script:
        lang: javascript
        id: process_args_filter
        file: process_args_filter.js
            
- module: system
  period: 30m
  metricsets:
    - cpu
  processors:
  - script:
        lang: javascript
        id: CPUStatusUpdater.js
        file: CPUStatusUpdater.js

Working metricbeat.yml for now:

metricbeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false
setup.ilm.enabled: true
setup.ilm.check_exists: true
setup.ilm.rollover_alias: cis-metric
setup.ilm.pattern: '{now/d}-000001'
setup.ilm.overwrite: false
setup.template.enabled: true  
setup.template.name: "cis-metric-%{[agent.version]}" 
setup.template.pattern: "cis-metric-%{[agent.version]}" 
setup.template.overwrite: false 
setup.ilm.policy_name: cis-metric
setup.template.settings:
  index.number_of_shards: 1
setup.kibana:
  host: https://abc:5601
output.elasticsearch:
  hosts: ["abc:100"]
  ssl.certificate_authorities: ["/l/app/cis/monitoring/config/ece_proxy_root_ca.pem"]
  protocol: ""
  username: ""
  password: ""
processors:
- rename:
    fields:
    - from: agent.hostname
      to: host.name
    ignore_missing: true
    fail_on_error: false
- drop_fields:
    fields:
    - host.mac
    - ecs
    - agent.ephemeral_id
    - agent.type
    - agent.version
    - agent.hostname
    - event

[stephenb (Stephen Brown)]

@PRASHANT_MEHTA

2)Real challenge now: I want to use metricbeat,filebeat and heartbeat(all time series data) now with my own CUSTOM templates following my own ILM policy defined.

Did you simply try to replace the generated ILM Policy and Template with yours with the same names etc after you ran setup?

@PRASHANT_MEHTA

Rollover are also not supported in 8.7.1 I guess? , and I use it in my template earlier.

Data streams rollover that is part of the core functionality... but you are right it is a bit different in 8.x Data Streams it is internal you do not use the write alias the Data Stream it IS the alias.

So what I did to test is I ran the setup with the config I had above.

Then you can just edit the ILM policy to what you want it to be
Edit the template to what you want it to be but there are a few key differences see below...

{
  "index_templates": [
    {
      "name": "cis-metric-8.8.0",
      "index_template": {
        "index_patterns": [
          "cis-metric-8.8.0" <!-- NOTE IMPORTANT The name of the Data Stream
        ],
        "template": {
          "settings": {
            "index": {
              "lifecycle": {
                "name": "cis-metric"
              },
              "codec": "best_compression",
              "mapping": {
                "total_fields": {
                  "limit": "10000"
                }
              },
              "refresh_interval": "5s",
              "number_of_shards": "1",
              "max_docvalue_fields_search": "200",
              "query": {
                "default_field": [
                  "message",
                  "tags"
                ]
              }
            }
          },
          "mappings": {
            "_meta": {
              "beat": "metricbeat",
              "version": "8.8.0"
            },
            "dynamic_templates": [
              ......
              ],
            "date_detection": false,
            "properties": {
              ....... Mappings Here
            }
          }
        },
        "composed_of": [],
        "priority": 150,
        "data_stream": {  <!--- NOTE IMPORTANT SUPER IMPORTANT I THINK YOU ARE MISSING THIS
          "hidden": false,
          "allow_custom_routing": false
        }
      }
    }
  ]
}

Then my metricbeat.yml looks like this...

setup.ilm.enabled: false
setup.template.enabled: false
output.elasticsearch:
  hosts: ["https://localhost:9200"]
  index: "cis-metric-%{[agent.version]}"   
  username: "elastic"
  password: "password"
  ssl.verification_mode: "none"

And now when I run metricbeat the data goes the correct places / data stream.

YOU can do all this manually as well setup you your entire data stream by following this

I like the setup method because it creates everything for me then I just edit and change to what I want

NOTE you are right you can run the setup with just this and it set up the ILM, Data Stream and Template the other stuff not needed any more

setup config

setup.ilm.enabled: true
setup.ilm.check_exists: true
setup.ilm.policy_name: cis-metric


setup.template.enabled: true  # <!--- You should explicitly set this 
setup.template.name: "cis-metric-%{[agent.version]}" # <!----This needs to match the index name because now it is a data stream
setup.template.pattern: "cis-metric-%{[agent.version]}" # <!-- Better matching Hygiene because it is a data stream 
setup.template.overwrite: false # <!--- Careful with this if set to true it will overwrite every time.

So run setup like that.

Then edit the ILM policy to what you want
Edit the Template Mappings to what you want and you are good...

If you save all that stuff you can skip the setup... just create them and then run with your custom data stream name.

run config

setup.ilm.enabled: false
setup.template.enabled: false
output.elasticsearch:
  hosts: ["https://localhost:9200"]
  index: "cis-metric-%{[agent.version]}"   
  username: "elastic"
  password: "password"
  ssl.verification_mode: "none"

hope this helps

[PRASHANT_MEHTA]

Hello @stephenb ,

I really appreciate your time and efforts to look into this and providing the needed settings to achieve my requirements. Many thanx for this!

I won't be able to test this today, but I will follow what you have recommended.
Just to calrify,If I'm getting you right.

Two methods:(Both methods assumes ILM is already defined)
1**)setup method**(setup config):
step1: commit the custom template
step2: run setup config(provided above)
Note in this method: setup.ilm.policy_name: cis-metric is provided in yml itself and in method two(run config) assumes ilm (index.lifecycle.name) to be part of template itself,from this it understands to pick ILM.

2)run config method(don't contain ilm settings in yml file and picks from template itself:index.lifecycle.name)
step1: commit the custom template
step 2:run setup config provided above

Will use the template you provided and edit just my mappings there.But i see this to be empty and do I need to fill something here, so that template works correct:

?

If this works then I'm assuming heartbeat and filebeat also follows the same approach?

Thanks

[stephenb (Stephen Brown)]

@PRASHANT_MEHTA

Two methods:(Both methods assumes ILM is already defined)
1**)setup method**(setup config):

No ... the setup method I showed you will create A basic ILM policy for you... you can edit it from there. If you run the setup I showed you it will create the cis-metrics ILM policy

@PRASHANT_MEHTA

Will use the template you provided and edit just my mappings there.But i see this to be empty and do I need to fill something here, so that template works correct:

Dynamic templates are up to you they are not required... you can read about them if you like, if you don't use them take them out...

[PRASHANT_MEHTA]

Hello @stephenb ,

Below is my custom index template and for this I tried two metricbeat.yml config and then started metricbeat.Now data stream was not created.

I hope the template you have shared was created custom and then tested?, bcz the syntax I see is of by deafult metricbeat in template and asked to add/edit additionaly1)my mappings 2)index patter anme,3)
"data_stream": {
"hidden": false,
"allow_custom_routing": false
}
Plz look my template and config metricbeat files.From what settings exaclty metricbeat will underastand to pick my own template,assuming below:
1st case:
setup.template.enabled: true
setup.template.name: "cis-metric-%{[agent.version]}"
setup.template.pattern: "cis-metric-%{[agent.version]}"

2nd case:(here ilm is not defined in what you have shared,then how it will understand which ILM)
setup.ilm.enabled: false
setup.template.enabled: false

I have changed in yml and in template the policy name to: cis-metric-policy as per my req.

Note: I'm building an ELK automation solution where in No user will ever come to Kibana page,
admins just vist custom webui integrated dashbords.
So what I do is,intially only create my cis-metric-policy with my settings in devtools/kibana policy page and then assumes when metricbeat will start with yml that also contains this policy name then it will automatically map to already created ILM policy.Intention to do so: No user will visit kibana page only webui.

PUT _index_template/cis-metric-8.7.1
{
        "index_patterns": [
          "cis-metric-8.7.1" 
        ],
        "template": {
          "settings": {
            "index": {
              "lifecycle": {
                "name": "cis-metric-policy"
              },
              "mapping": {
                "total_fields": {
                  "limit": "10000"
                }
              },
              "number_of_shards": "1"
            }
          },
          "mappings": {
            "properties": {
               "system.cpu.iowait.norm.pct": {
          "scaling_factor": 1000,
          "type": "scaled_float"
        },
        "system.process.memory.rss.pct": {
          "scaling_factor": 1000,
          "type": "scaled_float"
        },
        "system.diskio.write.bytes": {
          "type": "long"
        },
        "system.load.cores": {
          "type": "long"
        },
        "system.process.cpu.total.value": {
          "type": "long"
        },
        "service.type": {
          "type": "keyword"
        },
        "system.diskio.iostat.service_time": {
          "type": "float"
        },
        "system.memory.actual.free": {
          "type": "long"
        },
        "system.cpu.idle.norm.pct": {
          "scaling_factor": 1000,
          "type": "scaled_float"
        },
        "agent.name": {
          "type": "keyword"
        },
        "system.memory.total": {
          "type": "long"
        },
        "system.diskio.write.time": {
          "type": "long"
        },
        "process.ppid": {
          "type": "long"
        },
        "system.diskio.iostat.write.await": {
          "type": "float"
        },
        "system.fsstat.total_size.free": {
          "type": "long"
        },
        "system.memory.free": {
          "type": "long"
        },
        "system.memory.swap.free": {
          "type": "long"
        },
        "system.memory.actual.used.pct": {
          "scaling_factor": 100,
          "type": "scaled_float"
        },
        "system.diskio.iostat.queue.avg_size": {
          "type": "float"
        },
        "agent.id": {
          "type": "keyword"
        },
        "system.memory.actual.used.bytes": {
          "type": "long"
        },
        "system.diskio.iostat.busy": {
          "type": "float"
        },
        "system.diskio.name": {
          "type": "keyword"
        },
        "system.cpu.user.pct": {
          "scaling_factor": 1000,
          "type": "scaled_float"
        },
        "system.memory.used.bytes": {
          "type": "long"
        },
        "system.network.name": {
          "type": "keyword"
        },
        "system.diskio.iostat.request.avg_size": {
          "type": "float"
        },
        "system.diskspace.usage": {
          "type": "long"
        },
        "system.cpu.system.pct": {
          "scaling_factor": 1000,
          "type": "scaled_float"
        },
        "system.diskio.iostat.read.request.merges_per_sec": {
          "type": "float"
        },
        "system.cpu.irq.pct": {
          "scaling_factor": 1000,
          "type": "scaled_float"
        },
        "process.type": {
          "type": "keyword"
        },
        "process.details": {
          "type": "keyword"
        },
        "system.cpu.user.norm.pct": {
          "scaling_factor": 1000,
          "type": "scaled_float"
        },
        "user.name": {
          "type": "text",
          "fields": {
            "keyword": {
              "ignore_above": 256,
              "type": "keyword"
            }
          }
        },
        "system.network.in.bytes": {
          "type": "long"
        },
        "system.network.in.dropped": {
          "type": "long"
        },
        "system.diskio.io.ops": {
          "type": "long"
        },
        "mountpoint.details": {
          "type": "keyword"
        },
        "system.diskio.iostat.write.per_sec.bytes": {
          "type": "float"
        },
        "system.fsstat.count": {
          "type": "long"
        },
        "system.load.1": {
          "scaling_factor": 100,
          "type": "scaled_float"
        },
        "system.process.cpu.total.ticks": {
          "type": "long"
        },
        "system.cpu.softirq.norm.pct": {
          "scaling_factor": 1000,
          "type": "scaled_float"
        },
        "system.cpu.nice.pct": {
          "scaling_factor": 1000,
          "type": "scaled_float"
        },
        "system.load.5": {
          "scaling_factor": 100,
          "type": "scaled_float"
        },
        "metricset.period": {
          "type": "long"
        },
        "system.diskio.iostat.read.per_sec.bytes": {
          "type": "float"
        },
        "system.process.memory.share": {
          "type": "long"
        },
        "process.pgid": {
          "type": "long"
        },
        "system.process.memory.rss.bytes": {
          "type": "long"
        },
        "@timestamp": {
          "type": "date"
        },
        "system.network.in.errors": {
          "type": "long"
        },
        "system.process.cpu.total.norm.pct": {
          "scaling_factor": 1000,
          "type": "scaled_float"
        },
        "system.cpu.nice.norm.pct": {
          "scaling_factor": 1000,
          "type": "scaled_float"
        },
        "system.memory.used.pct": {
          "scaling_factor": 1000,
          "type": "scaled_float"
        },
        "system.diskio.iostat.read.await": {
          "type": "float"
        },
        "system.cpu.irq.norm.pct": {
          "scaling_factor": 1000,
          "type": "scaled_float"
        },
        "process.pid": {
          "type": "long"
        },
        "system.load.norm.15": {
          "scaling_factor": 100,
          "type": "scaled_float"
        },
        "system.diskio.read.count": {
          "type": "long"
        },
        "system.filesystem.used.pct": {
          "type": "float"
        },
        "system.cpu.iowait.pct": {
          "scaling_factor": 1000,
          "type": "scaled_float"
        },
        "system.process.memory.size": {
          "type": "long"
        },
        "system.network.out.dropped": {
          "type": "long"
        },
        "cpu.status": {
          "type": "keyword"
        },
        "host.name": {
          "type": "text",
          "fields": {
            "keyword": {
              "ignore_above": 256,
              "type": "keyword"
            }
          }
        },
        "system.diskio.iostat.await": {
          "type": "float"
        },
        "system.network.out.packets": {
          "type": "long"
        },
        "system.cpu.steal.norm.pct": {
          "scaling_factor": 1000,
          "type": "scaled_float"
        },
        "system.cpu.steal.pct": {
          "scaling_factor": 1000,
          "type": "scaled_float"
        },
        "system.diskio.write.count": {
          "type": "long"
        },
        "process.name": {
          "type": "text",
          "fields": {
            "keyword": {
              "ignore_above": 256,
              "type": "keyword"
            }
          }
        },
        "system.diskio.read.time": {
          "type": "long"
        },
        "system.diskio.serial_number": {
          "type": "keyword"
        },
        "system.fsstat.total_files": {
          "type": "long"
        },
        "system.network.in.packets": {
          "type": "long"
        },
        "system.cpu.total.pct": {
          "scaling_factor": 1000,
          "type": "scaled_float"
        },
        "system.network.out.errors": {
          "type": "long"
        },
        "system.diskio.iostat.write.request.merges_per_sec": {
          "type": "float"
        },
        "system.memory.swap.total": {
          "type": "long"
        },
        "system.diskio.io.time": {
          "type": "long"
        },
        "system.cpu.total.norm.pct": {
          "scaling_factor": 1000,
          "type": "scaled_float"
        },
        "system.load.15": {
          "scaling_factor": 100,
          "type": "scaled_float"
        },
        "system.cpu.cores": {
          "type": "long"
        },
        "system.fsstat.status": {
          "type": "keyword"
        },
        "system.cpu.softirq.pct": {
          "scaling_factor": 1000,
          "type": "scaled_float"
        },
        "system.load.norm.5": {
          "scaling_factor": 100,
          "type": "scaled_float"
        },
        "system.diskio.iostat.write.request.per_sec": {
          "type": "float"
        },
        "system.diskio.read.bytes": {
          "type": "long"
        },
        "system.fsstat.total_size.used": {
          "type": "long"
        },
        "system.load.norm.1": {
          "scaling_factor": 100,
          "type": "scaled_float"
        },
        "system.process.cpu.start_time": {
          "type": "date"
        },
        "system.memory.swap.used.pct": {
          "scaling_factor": 1000,
          "type": "scaled_float"
        },
        "system.cpu.idle.pct": {
          "scaling_factor": 1000,
          "type": "scaled_float"
        },
        "system.diskio.iostat.read.request.per_sec": {
          "type": "float"
        },
        "system.network.out.bytes": {
          "type": "long"
        },
        "system.memory.swap.used.bytes": {
          "type": "long"
        },
        "process.args": {
          "type": "keyword"
        },
        "system.cpu.system.norm.pct": {
          "scaling_factor": 1000,
          "type": "scaled_float"
        },
        "system.process.cpu.total.pct": {
          "scaling_factor": 1000,
          "type": "scaled_float"
        },
        "metricset.name": {
          "type": "keyword"
        },
        "system.fsstat.total_size.total": {
          "type": "long"
        },
        "system.process.state": {
          "type": "keyword"
        }
            }
          }
        },
        "composed_of": [],
        "data_stream": {  
          "hidden": false,
          "allow_custom_routing": false
        }
}

metricbeat.yml config 1:

metricbeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false
setup.ilm.enabled: true
setup.ilm.check_exists: true
setup.ilm.policy_name: cis-metric-policy
setup.template.enabled: true 
setup.template.name: "cis-metric-%{[agent.version]}" 
setup.template.pattern: "cis-metric-%{[agent.version]}"  
setup.template.overwrite: false 
setup.kibana:
  host: https://abc:5601
output.elasticsearch:
  hosts: ["https://abc:443"]
  index: "cis-metric-%{[agent.version]}"
  ssl.certificate_authorities: ["/l/ppp/config/ece_proxy_root_ca.pem"]
  protocol: "https"
  username: ""
  password: ""
processors:
- rename:
    fields:
    - from: agent.hostname
      to: host.name
    ignore_missing: true
    fail_on_error: false
- drop_fields:
    fields:
    - host.mac
    - ecs
    - agent.ephemeral_id
    - agent.type
    - agent.version
    - agent.hostname
    - event

Above congig didn't work then tried this:

metricbeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false
setup.ilm.enabled: false
setup.template.enabled: false 
setup.kibana:
  host: https://abc:5601
output.elasticsearch:
  hosts: ["https://abc:443"]
  index: "cis-metric-%{[agent.version]}"
  ssl.certificate_authorities: ["/l/ppp/config/ece_proxy_root_ca.pem"]
  protocol: "https"
  username: ""
  password: ""
processors:
- rename:
    fields:
    - from: agent.hostname
      to: host.name
    ignore_missing: true
    fail_on_error: false
- drop_fields:
    fields:
    - host.mac
    - ecs
    - agent.ephemeral_id
    - agent.type
    - agent.version
    - agent.hostname
    - event

Both ways data stream was not created.

[stephenb (Stephen Brown)]

Hi @PRASHANT_MEHTA

I get your requirement....I think you're missing my point.
If you run setup the way I showed you, it gives you all the correct assets... Best Practice assets (ILM, policy, template, data stream etc) it was just a way of getting a set of best practice assets.

Afterward, you just edit them to meet your requirements.
Then can you just capture those with a GET command (fix the opening and closing {}) fix mappings to what you want then POST and create the assets as part of your automation before you run metricbeat.

I think the confusion is that since you are creating all the assets MANUALLY you do not need to run setup and the datastream is created when you run metricbeat at runtime. not during setup ... which does not need to be run

Here is my complete code process using YOUR template (I changed the version number only)

First Clean Up and Post all the Assets ILM Policy and template

DELETE _data_stream/cis-metric-8.8.0

GET _ilm/policy/cis-metric-policy

DELETE _ilm/policy/cis-metric-policy

PUT _ilm/policy/cis-metric-policy
{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {
          "rollover": {
            "max_age": "30d",
            "max_primary_shard_size": "50gb"
          }
        }
      }
    }
  }
}

DELETE _index_template/cis-metric-8.8.0

# NOTE THIS IS YOUR TEMPLATE WITH JUST THE VERSION CHANGED

PUT _index_template/cis-metric-8.8.0
{
  "index_patterns": [
    "cis-metric-8.8.0"
  ],
  "template": {
    "settings": {
      "index": {
        "lifecycle": {
          "name": "cis-metric-policy"
        },
        "mapping": {
          "total_fields": {
            "limit": "10000"
          }
        },
        "number_of_shards": "1"
      }
    },
    "mappings": {
      "properties": {
        "system.cpu.iowait.norm.pct": {
          "scaling_factor": 1000,
          "type": "scaled_float"
        },
        "system.process.memory.rss.pct": {
          "scaling_factor": 1000,
          "type": "scaled_float"
        },
        "system.diskio.write.bytes": {
          "type": "long"
        },
        "system.load.cores": {
          "type": "long"
        },
        "system.process.cpu.total.value": {
          "type": "long"
        },
        "service.type": {
          "type": "keyword"
        },
        "system.diskio.iostat.service_time": {
          "type": "float"
        },
        "system.memory.actual.free": {
          "type": "long"
        },
        "system.cpu.idle.norm.pct": {
          "scaling_factor": 1000,
          "type": "scaled_float"
        },
        "agent.name": {
          "type": "keyword"
        },
        "system.memory.total": {
          "type": "long"
        },
        "system.diskio.write.time": {
          "type": "long"
        },
        "process.ppid": {
          "type": "long"
        },
        "system.diskio.iostat.write.await": {
          "type": "float"
        },
        "system.fsstat.total_size.free": {
          "type": "long"
        },
        "system.memory.free": {
          "type": "long"
        },
        "system.memory.swap.free": {
          "type": "long"
        },
        "system.memory.actual.used.pct": {
          "scaling_factor": 100,
          "type": "scaled_float"
        },
        "system.diskio.iostat.queue.avg_size": {
          "type": "float"
        },
        "agent.id": {
          "type": "keyword"
        },
        "system.memory.actual.used.bytes": {
          "type": "long"
        },
        "system.diskio.iostat.busy": {
          "type": "float"
        },
        "system.diskio.name": {
          "type": "keyword"
        },
        "system.cpu.user.pct": {
          "scaling_factor": 1000,
          "type": "scaled_float"
        },
        "system.memory.used.bytes": {
          "type": "long"
        },
        "system.network.name": {
          "type": "keyword"
        },
        "system.diskio.iostat.request.avg_size": {
          "type": "float"
        },
        "system.diskspace.usage": {
          "type": "long"
        },
        "system.cpu.system.pct": {
          "scaling_factor": 1000,
          "type": "scaled_float"
        },
        "system.diskio.iostat.read.request.merges_per_sec": {
          "type": "float"
        },
        "system.cpu.irq.pct": {
          "scaling_factor": 1000,
          "type": "scaled_float"
        },
        "process.type": {
          "type": "keyword"
        },
        "process.details": {
          "type": "keyword"
        },
        "system.cpu.user.norm.pct": {
          "scaling_factor": 1000,
          "type": "scaled_float"
        },
        "user.name": {
          "type": "text",
          "fields": {
            "keyword": {
              "ignore_above": 256,
              "type": "keyword"
            }
          }
        },
        "system.network.in.bytes": {
          "type": "long"
        },
        "system.network.in.dropped": {
          "type": "long"
        },
        "system.diskio.io.ops": {
          "type": "long"
        },
        "mountpoint.details": {
          "type": "keyword"
        },
        "system.diskio.iostat.write.per_sec.bytes": {
          "type": "float"
        },
        "system.fsstat.count": {
          "type": "long"
        },
        "system.load.1": {
          "scaling_factor": 100,
          "type": "scaled_float"
        },
        "system.process.cpu.total.ticks": {
          "type": "long"
        },
        "system.cpu.softirq.norm.pct": {
          "scaling_factor": 1000,
          "type": "scaled_float"
        },
        "system.cpu.nice.pct": {
          "scaling_factor": 1000,
          "type": "scaled_float"
        },
        "system.load.5": {
          "scaling_factor": 100,
          "type": "scaled_float"
        },
        "metricset.period": {
          "type": "long"
        },
        "system.diskio.iostat.read.per_sec.bytes": {
          "type": "float"
        },
        "system.process.memory.share": {
          "type": "long"
        },
        "process.pgid": {
          "type": "long"
        },
        "system.process.memory.rss.bytes": {
          "type": "long"
        },
        "@timestamp": {
          "type": "date"
        },
        "system.network.in.errors": {
          "type": "long"
        },
        "system.process.cpu.total.norm.pct": {
          "scaling_factor": 1000,
          "type": "scaled_float"
        },
        "system.cpu.nice.norm.pct": {
          "scaling_factor": 1000,
          "type": "scaled_float"
        },
        "system.memory.used.pct": {
          "scaling_factor": 1000,
          "type": "scaled_float"
        },
        "system.diskio.iostat.read.await": {
          "type": "float"
        },
        "system.cpu.irq.norm.pct": {
          "scaling_factor": 1000,
          "type": "scaled_float"
        },
        "process.pid": {
          "type": "long"
        },
        "system.load.norm.15": {
          "scaling_factor": 100,
          "type": "scaled_float"
        },
        "system.diskio.read.count": {
          "type": "long"
        },
        "system.filesystem.used.pct": {
          "type": "float"
        },
        "system.cpu.iowait.pct": {
          "scaling_factor": 1000,
          "type": "scaled_float"
        },
        "system.process.memory.size": {
          "type": "long"
        },
        "system.network.out.dropped": {
          "type": "long"
        },
        "cpu.status": {
          "type": "keyword"
        },
        "host.name": {
          "type": "text",
          "fields": {
            "keyword": {
              "ignore_above": 256,
              "type": "keyword"
            }
          }
        },
        "system.diskio.iostat.await": {
          "type": "float"
        },
        "system.network.out.packets": {
          "type": "long"
        },
        "system.cpu.steal.norm.pct": {
          "scaling_factor": 1000,
          "type": "scaled_float"
        },
        "system.cpu.steal.pct": {
          "scaling_factor": 1000,
          "type": "scaled_float"
        },
        "system.diskio.write.count": {
          "type": "long"
        },
        "process.name": {
          "type": "text",
          "fields": {
            "keyword": {
              "ignore_above": 256,
              "type": "keyword"
            }
          }
        },
        "system.diskio.read.time": {
          "type": "long"
        },
        "system.diskio.serial_number": {
          "type": "keyword"
        },
        "system.fsstat.total_files": {
          "type": "long"
        },
        "system.network.in.packets": {
          "type": "long"
        },
        "system.cpu.total.pct": {
          "scaling_factor": 1000,
          "type": "scaled_float"
        },
        "system.network.out.errors": {
          "type": "long"
        },
        "system.diskio.iostat.write.request.merges_per_sec": {
          "type": "float"
        },
        "system.memory.swap.total": {
          "type": "long"
        },
        "system.diskio.io.time": {
          "type": "long"
        },
        "system.cpu.total.norm.pct": {
          "scaling_factor": 1000,
          "type": "scaled_float"
        },
        "system.load.15": {
          "scaling_factor": 100,
          "type": "scaled_float"
        },
        "system.cpu.cores": {
          "type": "long"
        },
        "system.fsstat.status": {
          "type": "keyword"
        },
        "system.cpu.softirq.pct": {
          "scaling_factor": 1000,
          "type": "scaled_float"
        },
        "system.load.norm.5": {
          "scaling_factor": 100,
          "type": "scaled_float"
        },
        "system.diskio.iostat.write.request.per_sec": {
          "type": "float"
        },
        "system.diskio.read.bytes": {
          "type": "long"
        },
        "system.fsstat.total_size.used": {
          "type": "long"
        },
        "system.load.norm.1": {
          "scaling_factor": 100,
          "type": "scaled_float"
        },
        "system.process.cpu.start_time": {
          "type": "date"
        },
        "system.memory.swap.used.pct": {
          "scaling_factor": 1000,
          "type": "scaled_float"
        },
        "system.cpu.idle.pct": {
          "scaling_factor": 1000,
          "type": "scaled_float"
        },
        "system.diskio.iostat.read.request.per_sec": {
          "type": "float"
        },
        "system.network.out.bytes": {
          "type": "long"
        },
        "system.memory.swap.used.bytes": {
          "type": "long"
        },
        "process.args": {
          "type": "keyword"
        },
        "system.cpu.system.norm.pct": {
          "scaling_factor": 1000,
          "type": "scaled_float"
        },
        "system.process.cpu.total.pct": {
          "scaling_factor": 1000,
          "type": "scaled_float"
        },
        "metricset.name": {
          "type": "keyword"
        },
        "system.fsstat.total_size.total": {
          "type": "long"
        },
        "system.process.state": {
          "type": "keyword"
        }
      }
    }
  },
  "composed_of": [],
  "data_stream": {
    "hidden": false,
    "allow_custom_routing": false
  }
}

Then this is my metricbeat config... all that is needed for the template, IILM and output section (I am not looking at all your other config.

setup.ilm.enabled: false
setup.template.enabled: false
output.elasticsearch:
  hosts: ["https://localhost:9200"]
  index: "cis-metric-%{[agent.version]}"   
  username: "elastic"
  password: "password"
  ssl.verification_mode: "none"

AND I think this is your confusion since you set everything up Manually you do NOT need to run setup command the datastream is created at first run / run time

Then simply run
metricbeat -e
Or however, you want to run and the Data Stream is created at run time using the assets you previously installed.

That is all I did and I used your template

GET _data_stream/cis-metric-8.8.0

{
  "data_streams": [
    {
      "name": "cis-metric-8.8.0",
      "timestamp_field": {
        "name": "@timestamp"
      },
      "indices": [
        {
          "index_name": ".ds-cis-metric-8.8.0-2023.07.10-000001",
          "index_uuid": "dUKP92isQX2DUHbM4CaX7A"
        }
      ],
      "generation": 1,
      "status": "YELLOW",
      "template": "cis-metric-8.8.0",
      "ilm_policy": "cis-metric-policy",
      "hidden": false,
      "system": false,
      "allow_custom_routing": false,
      "replicated": false
    }
  ]
}

[PRASHANT_MEHTA]

@stephenb ,

Thanx again for your time to look into this. I will do whatever you recommended above and may test in few hours.Thanx for your patience ,Migration is hectic.
Only confusion: Where the datastream will understand which policy to follow for cis-metric-8.7.1,as in below config ilm policy info is not there,even what I do PUT ILM policy and template itself first in devtools then run metricbeat.I'm thinking how the data stream will get to know which policy to follow with below:

setup.ilm.enabled: false
setup.template.enabled: false
output.elasticsearch:
  hosts: ["https://localhost:9200"]
  index: "cis-metric-%{[agent.version]}"   
  username: "elastic"
  password: "password"
  ssl.verification_mode: "none"

If you understood me right,what I do:
1)First commit template and ILM in devtools
2)run metricbrat.yml
3)Expect that data stream will now map with my template only and follow my ILM only(commited earlier in dev tools)

How metric beat is run by me:

start.sh

#!/bin/ksh
echo "Starting metricbeat..."
./metricbeat & 
_pid=$! 
echo "$_pid" > metricbeat.pid
echo "Pid $_pid stored in metricbeat.pid"

[stephenb (Stephen Brown)]

@PRASHANT_MEHTA

Only confusion: Where the datastream will understand which policy to follow for cis-metric-8.7.1,as in below config ilm policy info is not there,even what I do PUT ILM policy and template itself first in devtools then run metricbeat.

It is defined in the template that is all that is needed...
Yes if you put the ILM policy and the template in FIRST before running metricbeat then it should work.

PUT _index_template/cis-metric-8.8.0
{
  "index_patterns": [
    "cis-metric-8.8.0"
  ],
  "template": {
    "settings": {
      "index": {
        "lifecycle": {
          "name": "cis-metric-policy" <!---- Right Here
        },

BTW all the manual setup (i.e. no running setup ) is actually a very common approach for customized deployments.

[PRASHANT_MEHTA]

Hello @stephenb ,

Many thanx for your time on this.Everytime I wondered same index template and ILM I provided you and things work fine with you and not my end.
Now issue is resolved but please be informed how its working now and not previously.

How issue got solved:

As you mentioned you are using 8.8.0 ,here this is valid(cis-metric-%{[agent.version]}),same is not valid in 8.7.1 and I need to manually put index name only and now it works,weird:

8.8.0:

index: "cis-metric-%{[agent.version]}"(yml)

I'm using 8.7.1 version and I need to do below,this is not good practice,any alterantive or what might be the reason in 8.7.1 this cis-metric-%{[agent.version]} dont work:
8.7.1 using below then only data stream created.
index: "cis-metric-8.7.1"(yml)

Alternatively do u think it would be fine if i remove 8.7.1 from-index pattern,template and from yml and instead only use cis-metric only.

I hope similar is the approach for all beats i.e heartbeat and filebeat(custom template and ilm),like how did for metricbeat for data streams?