# Index Lifecycle Management Metricbeat

**URL:** <https://discuss.elastic.co/t/index-lifecycle-management-metricbeat/318757>\
**Category:** Elasticsearch\
**Created:** [November 11, 2022, 5:53pm UTC](https://discuss.elastic.co/t/index-lifecycle-management-metricbeat/318757 "2022-11-11T17:53:41Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![Ankita\_Pachauri](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ankita_pachauri/32/61469_2.png) [@Ankita\_Pachauri](https://discuss.elastic.co/u/Ankita_Pachauri)\
**Post date:** [November 11, 2022, 5:53pm UTC](https://discuss.elastic.co/t/index-lifecycle-management-metricbeat/318757/1 "2022-11-11T17:53:41Z")

</div>

Hi Team,  
I have metricbeat installed on my servers to send data to ES. I want to have daily indexes for each day. Along with this, I also want to delete the indexes after few days using ILM. I am attaching my metricbeat.yml configuration.

Note: I have 3 ES nodes as cluster and 2 nodes for Logstash and Kibana.

```auto
> ###################### Metricbeat Configuration Example #######################
> 
> # This file is an example configuration file highlighting only the most common
> # options. The metricbeat.reference.yml file from the same directory contains all the
> # supported options with more comments. You can use it as a reference.
> #
> # You can find the full configuration reference here:
> # https://www.elastic.co/guide/en/beats/metricbeat/index.html
> 
> # =========================== Modules configuration ============================
> 
> metricbeat.config.modules:
> # Glob pattern for configuration loading
> path: ${path.config}/modules.d/*.yml
> 
> # Set to true to enable config reloading
> reload.enabled: false
> 
> # Period on which files under path should be checked for changes
> #reload.period: 10s
> 
> # ======================= Elasticsearch template setting =======================
> setup.template.name: "metric"
> setup.template.fields: "fields.yml"
> setup.template.overwrite: true
> setup.template.pattern: "metricbeateoprodsec-*"
> setup.template.settings:
> index.number_of_shards: 3
> index.number_of_replicas: 1
> index.codec: best_compression
> #_source.enabled: false
> 
> 
> # ================================== General ===================================
> 
> # The name of the shipper that publishes the network data. It can be used to group
> # all the transactions sent by a single shipper in the web interface.
> #name:
> 
> # The tags of the shipper are included in their own field with each
> # transaction published.
> #tags: ["service-X", "web-tier"]
> 
> # Optional fields that you can specify to add additional information to the
> # output.
> #fields:
> # env: staging
> 
> # ================================= Dashboards =================================
> # These settings control loading the sample dashboards to the Kibana index. Loading
> # the dashboards is disabled by default and can be enabled either by setting the
> # options here or by using the `setup` command.
> #setup.dashboards.enabled: true
> 
> # The URL from where to download the dashboards archive. By default this URL
> # has a value which is computed based on the Beat name and version. For released
> # versions, this URL points to the dashboard archive on the artifacts.elastic.co
> # website.
> #setup.dashboards.url:
> 
> # =================================== Kibana ===================================
> 
> # Starting with Beats version 6.0.0, the dashboards are loaded via the Kibana API.
> # This requires a Kibana endpoint configuration.
> setup.kibana:
> 
> # Kibana Host
> # Scheme and port can be left out and will be set to the default (http and 5601)
> # In case you specify and additional path, the scheme is required: http://localhost:5601/path
> # IPv6 addresses should always be defined as: https://[2001:db8::1]:5601
> host: "http://cms-public-nlb-bae35f5967957d10.elb.us-east-1.amazonaws.com/:5601"
> username: "admin"
> password: "admin"
> 
> # Kibana Space ID
> # ID of the Kibana Space into which the dashboards should be loaded. By default,
> # the Default Space will be used.
> #space.id:
> 
> # =============================== Elastic Cloud ================================
> 
> # These settings simplify using Metricbeat with the Elastic Cloud (https://cloud.elastic.co/).
> 
> # The cloud.id setting overwrites the `output.elasticsearch.hosts` and
> # `setup.kibana.host` options.
> # You can find the `cloud.id` in the Elastic Cloud web UI.
> #cloud.id:
> 
> # The cloud.auth setting overwrites the `output.elasticsearch.username` and
> # `output.elasticsearch.password` settings. The format is `<user>:<pass>`.
> #cloud.auth:
> 
> # ================================== Outputs ===================================
> 
> # Configure what output to use when sending the data collected by the beat.
> 
> # ---------------------------- Elasticsearch Output ----------------------------
> output.elasticsearch:
> # Array of hosts to connect to.
> hosts: ["https://cms-public-nlb-bae35f5967957d10.elb.us-east-1.amazonaws.com:9200"]
> indices:
> - index: "metricbeateoprodsec-%{[agent.version]}-%{+yyyy.MM.dd}"
> # Protocol - either `http` (default) or `https`.
> protocol: "https"
> 
> # Authentication credentials - either API key or username/password.
> #api_key: "id:api_key"
> username: "admin"
> password: "admin"
> ssl.verification_mode: none
> 
> # ------------------------------ Logstash Output -------------------------------
> #output.logstash:
> # The Logstash hosts
> #hosts: ["localhost:5044"]
> 
> # Optional SSL. By default is off.
> # List of root certificates for HTTPS server verifications
> #ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]
> 
> # Certificate for SSL client authentication
> #ssl.certificate: "/etc/pki/client/cert.pem"
> 
> # Client Certificate Key
> #ssl.key: "/etc/pki/client/cert.key"
> 
> # ================================= Processors =================================
> 
> # Configure processors to enhance or manipulate events generated by the beat.
> 
> processors:
> - add_host_metadata: ~
> - add_cloud_metadata: ~
> - add_docker_metadata: ~
> - add_kubernetes_metadata: ~
> 
> 
> # ================================== Logging ===================================
> 
> # Sets log level. The default log level is info.
> # Available log levels are: error, warning, info, debug
> #logging.level: debug
> 
> # At debug level, you can selectively enable logging only for some components.
> # To enable all selectors use ["*"]. Examples of other selectors are "beat",
> # "publisher", "service".
> #logging.selectors: ["*"]
> 
> # ============================= X-Pack Monitoring ==============================
> # Metricbeat can export internal metrics to a central Elasticsearch monitoring
> # cluster. This requires xpack monitoring to be enabled in Elasticsearch. The
> # reporting is disabled by default.
> 
> # Set to true to enable the monitoring reporter.
> #monitoring.enabled: false
> 
> # Sets the UUID of the Elasticsearch cluster under which monitoring data for this
> # Metricbeat instance will appear in the Stack Monitoring UI. If output.elasticsearch
> # is enabled, the UUID is derived from the Elasticsearch cluster referenced by output.elasticsearch.
> #monitoring.cluster_uuid:
> 
> # Uncomment to send the metrics to Elasticsearch. Most settings from the
> # Elasticsearch output are accepted here as well.
> # Note that the settings should point to your Elasticsearch *monitoring* cluster.
> # Any setting that is not set is automatically inherited from the Elasticsearch
> # output configuration, so if you have the Elasticsearch output configured such
> # that it is pointing to your Elasticsearch monitoring cluster, you can simply
> # uncomment the following line.
> #monitoring.elasticsearch:
> 
> # ============================== Instrumentation ===============================
> 
> # Instrumentation support for the metricbeat.
> #instrumentation:
> # Set to true to enable instrumentation of metricbeat.
> #enabled: false
> 
> # Environment in which metricbeat is running on (eg: staging, production, etc.)
> #environment: ""
> 
> # APM Server hosts to report instrumentation results to.
> #hosts:
> # - http://localhost:8200
> 
> # API Key for the APM Server(s).
> # If api_key is set then secret_token will be ignored.
> #api_key:
> 
> # Secret token for the APM Server(s).
> #secret_token:
> 
> 
> # ================================= Migration ==================================
> 
> # This allows to enable 6.7 migration aliases
> #migration.6_to_7.enabled: true

```

Kindly suggest.

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [November 12, 2022, 6:17pm UTC](https://discuss.elastic.co/t/index-lifecycle-management-metricbeat/318757/2 "2022-11-12T18:17:26Z")

</div>

Hi @Ankita_Pachauri

What version?

Below this is for 8.x but should work for 7.17 as well...

And I always add that although renaming default indices is supported however if you are just getting to know The Elastic Stack there are tons of reasons to just use the defaults, when you move away from the defaults it is critical you understand all those relationship.

Another approach is simply tagging these host(s) with a tag  
`tags: ["metricbeateoprodsec"]`

and leave everything as default and just filter by the tag in each of the areas and set the metric ILM policy to what you want

These are all the things affected when changing index / data stream names

- Data Stream
- Index Template
- ILM Policy
- Dashboards
- Infrastructure App.
- Probably more I am not remembering

So assuming you want to do this...Leverage the framework

Here is my fully functional metricbeat.yml

You will want to run setup Once then turn off the overwrite and dashboards loading.

Run setup first  
`filebeat setup -e`

then start filebeat  
`filebeat -e`  
or  
`systemctl start filebeat`

```auto
# =========================== Modules configuration ============================

metricbeat.config.modules:
  # Glob pattern for configuration loading
  path: ${path.config}/modules.d/*.yml

  # Set to true to enable config reloading
  reload.enabled: false

  # Period on which files under path should be checked for changes
  #reload.period: 10s

# ======================= Elasticsearch template setting =======================

setup.template.enabled: true  
setup.template.name: "metricbeateoprodsec-%{[agent.version]}" 
setup.template.pattern: "metricbeateoprodsec-%{[agent.version]}" 
setup.template.overwrite: true ## <- Set this to true Once then take out or set to false. 
setup.ilm.policy_name: metricbeateoprodsec-policy 
setup.template.settings:
  index.number_of_shards: 1
  index.number_of_replicas: 1
  index.codec: best_compression
  #_source.enabled: false

# Important if you want the dashboards to work
# Set this to true Once then set to false. Or comment out BOTH lines, Otherwise metricbeat will reload the dashboards EVERY TIME!
setup.dashboards.enabled: true 
setup.dashboards.index: "metricbeateoprodsec-*"

# =================================== Kibana ===================================

setup.kibana:

# ---------------------------- Elasticsearch Output ----------------------------
output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["http://localhost:9200"]
  index: "metricbeateoprodsec-%{[agent.version]}"

processors:
  - add_host_metadata: ~
  - add_cloud_metadata: ~
  - add_docker_metadata: ~
  - add_kubernetes_metadata: ~

```

Now you have a Data Stream, Index Template and ILM Policy and have set the Built in dashboards to work with all the changes

Data Stream

 ![Screen Shot 2022-11-12 at 10.04.12 AM](https://us1.discourse-cdn.com/elastic/original/3X/8/6/86be80a250242fcbcfa7b6676ea1d6f90192bf4d.png)

Index Template

 ![Screen Shot 2022-11-12 at 10.04.18 AM](https://us1.discourse-cdn.com/elastic/original/3X/8/4/84ceab1b0253a3d0b7fe9d8628de3951220786a0.png)

ILM Policy

 ![Screen Shot 2022-11-12 at 10.04.26 AM](https://us1.discourse-cdn.com/elastic/original/3X/b/7/b7b26a224243b66a0ae74db9660f188613428854.png)

Now for the ILM you have an ILM Policy go in and adjust as you wish

 ![Screen Shot 2022-11-12 at 10.06.49 AM](https://us1.discourse-cdn.com/elastic/original/3X/b/1/b128fbf6642a82eddcdaf2c16ba0e6d8d809062b.jpeg)

Don't forget to fix the Infrasture App (No Spaces)

 ![Screen Shot 2022-11-12 at 10.08.00 AM](https://us1.discourse-cdn.com/elastic/original/3X/7/8/7862453bfdc194b84735e95901f7b9c79305c7da.png)

Dashboards

 ![Screen Shot 2022-11-12 at 10.08.52 AM](https://us1.discourse-cdn.com/elastic/original/3X/7/1/71eaa40fcb16d5dc3f4332d41defb26bf26e320c.png)

---

<div class="post-metadata">

**Author:** ![Ankita\_Pachauri](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ankita_pachauri/32/61469_2.png) [@Ankita\_Pachauri](https://discuss.elastic.co/u/Ankita_Pachauri)\
**Post date:** [November 14, 2022, 5:58am UTC](https://discuss.elastic.co/t/index-lifecycle-management-metricbeat/318757/3 "2022-11-14T05:58:45Z")

</div>

Hi Stephan,  
Are the indices created date wise? I also want to have indices created date wise, it would be helpful in taking snapshot of the indices.

//Ankita

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [November 14, 2022, 6:19am UTC](https://discuss.elastic.co/t/index-lifecycle-management-metricbeat/318757/4 "2022-11-14T06:19:50Z")

</div>

Set it up and run

`GET _cat/indices/?v`

What do you see?

. or go into  
Kibana - Stack Management - Data Streams and click on the linked indices

Or just Stack Management - Indices and show hidden indices.

What do you see?

---

<div class="post-metadata">

**Author:** ![Ankita\_Pachauri](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ankita_pachauri/32/61469_2.png) [@Ankita\_Pachauri](https://discuss.elastic.co/u/Ankita_Pachauri)\
**Post date:** [November 27, 2022, 5:26pm UTC](https://discuss.elastic.co/t/index-lifecycle-management-metricbeat/318757/5 "2022-11-27T17:26:10Z")

</div>

Hi,  
I have mentioned the same file as mentioned by you, however I am only getting index as metricbeat-7.17.6 and not the string eoprod combined with it.

Please support

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [November 27, 2022, 6:03pm UTC](https://discuss.elastic.co/t/index-lifecycle-management-metricbeat/318757/6 "2022-11-27T18:03:45Z")

</div>

Hi @Ankita_Pachauri

Unless you show exactly what you did every single command, the command output and every configuration we can't help. Just saying it does not work does not provide enough information to help.

It does assume that you understand you need to run `filebeat setup -e` before starting filebeat.

The example I gave above works if you follow it very specifically.

Also, just to be clear, this is a community forum... It's not paid support so sometimes your questions may not be able to be answered.

---

<div class="post-metadata">

**Author:** ![Ankita\_Pachauri](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ankita_pachauri/32/61469_2.png) [@Ankita\_Pachauri](https://discuss.elastic.co/u/Ankita_Pachauri)\
**Post date:** [November 30, 2022, 5:58am UTC](https://discuss.elastic.co/t/index-lifecycle-management-metricbeat/318757/7 "2022-11-30T05:58:15Z")

</div>

Hi Stephen,  
I have followed the same metricbeat.yml file as mentioned by you. It has created a ILM policy as well, however the name of the indexes are as below:

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/b/8/b8fcf4e012ae008624ee06e90c0c7a4ba57a9690.png)

My requirement was that the name of the index should be metricbeateoprodsec-\*.

Please assist.

//Ankita

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [November 30, 2022, 6:11am UTC](https://discuss.elastic.co/t/index-lifecycle-management-metricbeat/318757/8 "2022-11-30T06:11:07Z")

</div>

Please show the exact metricbeat.yml and each command you ran.

What version are you on? Each component?

I ask these specific questions so that I can help.. other wise I can't help.

---

<div class="post-metadata">

**Author:** ![Ankita\_Pachauri](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ankita_pachauri/32/61469_2.png) [@Ankita\_Pachauri](https://discuss.elastic.co/u/Ankita_Pachauri)\
**Post date:** [November 30, 2022, 7:03pm UTC](https://discuss.elastic.co/t/index-lifecycle-management-metricbeat/318757/9 "2022-11-30T19:03:50Z")

</div>

Hi Stephan,  
Please find my yml file as:

```auto
###################### Metricbeat Configuration Example #######################

# This file is an example configuration file highlighting only the most common
# options. The metricbeat.reference.yml file from the same directory contains all the
# supported options with more comments. You can use it as a reference.
#
# You can find the full configuration reference here:
# https://www.elastic.co/guide/en/beats/metricbeat/index.html

# =========================== Modules configuration ============================

metricbeat.config.modules:
  # Glob pattern for configuration loading
  path: ${path.config}/modules.d/*.yml

  # Set to true to enable config reloading
  reload.enabled: false

  # Period on which files under path should be checked for changes
  #reload.period: 10s

# ======================= Elasticsearch template setting =======================

#setup.template.settings:
  #index.number_of_shards: 1
 # index.codec: best_compression
  #_source.enabled: false

setup.template.enabled: true
setup.template.name: "metricbeateoprod-%{[agent.version]}"
setup.template.pattern: "metricbeateoprod-%{[agent.version]}"
setup.template.overwrite: true ## <- Set this to true Once then take out or set to false.
setup.ilm.policy_name: metricbeateoprod-policy
setup.template.settings:
  index.number_of_shards: 3
  index.number_of_replicas: 1
  index.codec: best_compression
  #_source.enabled: false

# Important if you want the dashboards to work
# Set this to true Once then set to false. Or comment out BOTH lines, Otherwise metricbeat will reload the dashboards EVERY TIME!
setup.dashboards.enabled: true
setup.dashboards.index: "metricbeateoprod-*"

# ================================== General ===================================

# The name of the shipper that publishes the network data. It can be used to group
# all the transactions sent by a single shipper in the web interface.
#name:

# The tags of the shipper are included in their own field with each
# transaction published.
#tags: ["service-X", "web-tier"]

# Optional fields that you can specify to add additional information to the
# output.
#fields:
# env: staging

# ================================= Dashboards =================================
# These settings control loading the sample dashboards to the Kibana index. Loading
# the dashboards is disabled by default and can be enabled either by setting the
# options here or by using the `setup` command.
#setup.dashboards.enabled: false

# The URL from where to download the dashboards archive. By default this URL
# has a value which is computed based on the Beat name and version. For released
# versions, this URL points to the dashboard archive on the artifacts.elastic.co
# website.
#setup.dashboards.url:

# =================================== Kibana ===================================

# Starting with Beats version 6.0.0, the dashboards are loaded via the Kibana API.
# This requires a Kibana endpoint configuration.
setup.kibana:

  # Kibana Host
  # Scheme and port can be left out and will be set to the default (http and 5601)
  # In case you specify and additional path, the scheme is required: http://localhost:5601/path
  # IPv6 addresses should always be defined as: https://[2001:db8::1]:5601
  host: "http://xxx.xxx.xxx.xxx:5601"
  username: "admin"
  password: "admin"

  # Kibana Space ID
  # ID of the Kibana Space into which the dashboards should be loaded. By default,
  # the Default Space will be used.
  #space.id:

# =============================== Elastic Cloud ================================

# These settings simplify using Metricbeat with the Elastic Cloud (https://cloud.elastic.co/).

# The cloud.id setting overwrites the `output.elasticsearch.hosts` and
# `setup.kibana.host` options.
# You can find the `cloud.id` in the Elastic Cloud web UI.
#cloud.id:

# The cloud.auth setting overwrites the `output.elasticsearch.username` and
# `output.elasticsearch.password` settings. The format is `<user>:<pass>`.
#cloud.auth:

# ================================== Outputs ===================================

# Configure what output to use when sending the data collected by the beat.

# ---------------------------- Elasticsearch Output ----------------------------
output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["https://xxx.xxx.xxx.xxx:9200"]
  username: "admin"
  password: "admin"
  index: "metricbeateoprod-%{[agent.version]}"
  ssl.verification_mode: none

  # Protocol - either `http` (default) or `https`.
  protocol: "https"

  # Authentication credentials - either API key or username/password.
  #api_key: "id:api_key"
  #username: "elastic"
  #password: "changeme"

# ------------------------------ Logstash Output -------------------------------
#output.logstash:
  # The Logstash hosts
  #hosts: ["localhost:5044"]

  # Optional SSL. By default is off.
  # List of root certificates for HTTPS server verifications
  #ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]

  # Certificate for SSL client authentication
  #ssl.certificate: "/etc/pki/client/cert.pem"

  # Client Certificate Key
  #ssl.key: "/etc/pki/client/cert.key"

# ================================= Processors =================================

# Configure processors to enhance or manipulate events generated by the beat.

processors:
  - add_host_metadata: ~
  - add_cloud_metadata: ~
  - add_docker_metadata: ~
  - add_kubernetes_metadata: ~

# ================================== Logging ===================================

# Sets log level. The default log level is info.
# Available log levels are: error, warning, info, debug
#logging.level: debug

# At debug level, you can selectively enable logging only for some components.
# To enable all selectors use ["*"]. Examples of other selectors are "beat",
# "publisher", "service".
#logging.selectors: ["*"]

# ============================= X-Pack Monitoring ==============================
# Metricbeat can export internal metrics to a central Elasticsearch monitoring
# cluster. This requires xpack monitoring to be enabled in Elasticsearch. The
# reporting is disabled by default.

# Set to true to enable the monitoring reporter.
#monitoring.enabled: false

# Sets the UUID of the Elasticsearch cluster under which monitoring data for this
# Metricbeat instance will appear in the Stack Monitoring UI. If output.elasticsearch
# is enabled, the UUID is derived from the Elasticsearch cluster referenced by output.elasticsearch.
#monitoring.cluster_uuid:

# Uncomment to send the metrics to Elasticsearch. Most settings from the
# Elasticsearch output are accepted here as well.
# Note that the settings should point to your Elasticsearch *monitoring* cluster.
# Any setting that is not set is automatically inherited from the Elasticsearch
# output configuration, so if you have the Elasticsearch output configured such
# that it is pointing to your Elasticsearch monitoring cluster, you can simply
# uncomment the following line.
#monitoring.elasticsearch:

# ============================== Instrumentation ===============================

# Instrumentation support for the metricbeat.
#instrumentation:
    # Set to true to enable instrumentation of metricbeat.
    #enabled: false

    # Environment in which metricbeat is running on (eg: staging, production, etc.)
    #environment: ""

    # APM Server hosts to report instrumentation results to.
    #hosts:
    # - http://localhost:8200

    # API Key for the APM Server(s).
    # If api_key is set then secret_token will be ignored.
    #api_key:

    # Secret token for the APM Server(s).
    #secret_token:

# ================================= Migration ==================================

# This allows to enable 6.7 migration aliases
#migration.6_to_7.enabled: true

```

I am using ELK Stack 7.17.6 version.

After making changes to the yml file, i have restarted the metricbeat service after deleting the index template as well.

Please let me know in case you need more info.

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [November 30, 2022, 7:36pm UTC](https://discuss.elastic.co/t/index-lifecycle-management-metricbeat/318757/10 "2022-11-30T19:36:17Z")

</div>

What modules do you have enabled?

---

<div class="post-metadata">

**Author:** ![elasticforme](https://avatars.discourse-cdn.com/v4/letter/e/f05b48/32.png) [@elasticforme](https://discuss.elastic.co/u/elasticforme)\
**Post date:** [November 30, 2022, 7:40pm UTC](https://discuss.elastic.co/t/index-lifecycle-management-metricbeat/318757/11 "2022-11-30T19:40:53Z")

</div>

you are missing few stuff. as per my experience.

delete your index template, index, ILM and redo

```auto
output.elasticsearch.index: "metricbeateoprod-%{[agent.version]}"
setup.template.name: "metricbeateoprod-%{[agent.version]}"
setup.template.pattern: "metricbeateoprod-%{[agent.version]}"
setup.template.overwrite: true ## <- Set this to true Once then take out or set to false.
setup.template.enabled: true ## <- Set this to true Once then take out or set to false.

setup.ilm.policy_name: "metricbeateoprod-%{[agent.version]}"
setup.ilm.overwrite: true ## <- Set this to true Once then take out or set to false.
setup.ilm.enabled: true ## <- Set this to true Once then take out or set to false.
setup.ilm.policy_name: "metricbeateoprod-%{[agent.version]}"
setup.ilm.rollover_alias: "metricbeateoprod-%{[agent.version]}" ## this will be used to rollover to new index name as date-000002
setup.ilm.pattern: "{now/d}-000001" ## this is setup with date-000001 as first index name

```

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [November 30, 2022, 10:23pm UTC](https://discuss.elastic.co/t/index-lifecycle-management-metricbeat/318757/12 "2022-11-30T22:23:12Z")

</div>

@Ankita_Pachauri Apologies what I gave you was for 8.x ...

Let me get is correct for 7.x and reply when I can.

I think what @elasticforme Is saying is correct using his config run setup then turn set the setting he says to to false when actually runing

I will try to check later / tomorrow.

---

<div class="post-metadata">

**Author:** ![elasticforme](https://avatars.discourse-cdn.com/v4/letter/e/f05b48/32.png) [@elasticforme](https://discuss.elastic.co/u/elasticforme)\
**Post date:** [November 30, 2022, 10:41pm UTC](https://discuss.elastic.co/t/index-lifecycle-management-metricbeat/318757/13 "2022-11-30T22:41:20Z")

</div>

I used this setting first on 7.17.1.  
I used this last night on 8.5.2 and it worked.

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [November 30, 2022, 10:45pm UTC](https://discuss.elastic.co/t/index-lifecycle-management-metricbeat/318757/14 "2022-11-30T22:45:46Z")

</div>

But to be clear and precise

You run the config you show above as is  
`metricbeat setup -e`

Then set the settings you indicated to false to then start metricbeat

`metricbeat -e`  
or with  
`systemctl start metricbeat`

Correct / Confirm?

---

<div class="post-metadata">

**Author:** ![elasticforme](https://avatars.discourse-cdn.com/v4/letter/e/f05b48/32.png) [@elasticforme](https://discuss.elastic.co/u/elasticforme)\
**Post date:** [November 30, 2022, 11:00pm UTC](https://discuss.elastic.co/t/index-lifecycle-management-metricbeat/318757/15 "2022-11-30T23:00:19Z")

</div>

What I did is  
setup metricbeat.yml like this only on A system for initial setup.  
started metricbeat "systemctl start metricbeat"  
stop metricbeat "systemctl stop metricbeat"

edited metricbeat.yml, disable few line as required.

now all other metricbeat do not need any such thing. they just need output.index name ""metricbeateoprod-%{[agent.version]}" which is now alias to index ""metricbeateoprod-8.5.2-date-0000001"

and ILM will rollover when it reaches it limit.

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [November 30, 2022, 11:07pm UTC](https://discuss.elastic.co/t/index-lifecycle-management-metricbeat/318757/16 "2022-11-30T23:07:22Z")

</div>

@elasticforme I can see what you did will work but without running setup actually the dashboards and other assets will not be properly configured. (Example out of the box dashboards would not work without setup)

So I still think the correct steps is to use your configuration and specifically run setup.

Then edit those specific settings and then properly start metricbeat.

@Ankita_Pachauri

I will validate all this and reply back later or tomorrow.

Thank you @elasticforme

the 8.x data streams I think are a little easier.

---

<div class="post-metadata">

**Author:** ![elasticforme](https://avatars.discourse-cdn.com/v4/letter/e/f05b48/32.png) [@elasticforme](https://discuss.elastic.co/u/elasticforme)\
**Post date:** [November 30, 2022, 11:14pm UTC](https://discuss.elastic.co/t/index-lifecycle-management-metricbeat/318757/17 "2022-11-30T23:14:18Z")

</div>

oh ya. I do not use or load premade dashboard hence I do not use that setup steps.

yes I was so confuse when I did this for 8.5.2 as it created data stream and .ds-\* index and I can't find out the way it was working before.  
I was freaking out then suddenly show by clicking each button. 🤣

then read up about it and decided to test out this data stream part of it

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [December 1, 2022, 1:26am UTC](https://discuss.elastic.co/t/index-lifecycle-management-metricbeat/318757/18 "2022-12-01T01:26:44Z")

</div>

@Ankita_Pachauri Apologies this took too long it should not be this hard confusing (and I made it worse!)

Thanks @elasticforme For helping and below is the official method for changing the index names and still support OOTB dashboards etc.

Here is fully tested on 7.17.3

First you _must_ clean up all the metricbeat indices, index templates, ILM policies from previous tries.

Then you will use this config and run `setup`, You need to run `setup` it will take a while as it will reload all the dashboards.  
**NOTE:** `setup` only need to be run Once, not on every host, just 1 time for all hosts.

```auto
setup.ilm.enabled: true 
setup.ilm.overwrite: true ## <-- After Setup set to false
setup.ilm.policy_name: metricbeateoprodsec-policy 
setup.ilm.rollover_alias: "metricbeateoprodsec-%{[agent.version]}" 
setup.ilm.pattern: "{now/d}-000001"

setup.template.enabled: true
setup.template.overwrite: true ## <-- After Setup set to false
setup.template.name: "metricbeateoprodsec-%{[agent.version]}" 
setup.template.pattern: "metricbeateoprodsec-%{[agent.version]}-*" 

setup.template.settings:
  index.number_of_shards: 1
  index.number_of_replicas: 1
  index.codec: best_compression

setup.dashboards.enabled: true ## <-- After Setup set to false
setup.dashboards.index: "metricbeateoprodsec-*"

```

Using Config Above

`metricbeat setup -e`

Afterwards you should see the Index Template, The Index and the ILM Policy.

 ![Screen Shot 2022-11-30 at 5.17.31 PM](https://us1.discourse-cdn.com/elastic/original/3X/f/5/f57e32862750020b738853316e73a8a0948b7d3d.png)

 ![Screen Shot 2022-11-30 at 5.17.37 PM](https://us1.discourse-cdn.com/elastic/original/3X/a/9/a9d50980efc37a79618328609bbaee7dae6b62ef.png)

 ![Screen Shot 2022-11-30 at 5.17.46 PM](https://us1.discourse-cdn.com/elastic/original/3X/b/0/b076e2eb3c0bea7e695b5c2c09385f415d62cdca.png)

Then Update the config to...

```auto
setup.ilm.enabled: true 
setup.ilm.overwrite: false ## <-- After Setup set to false
setup.ilm.policy_name: metricbeateoprodsec-policy 
setup.ilm.rollover_alias: "metricbeateoprodsec-%{[agent.version]}" 
setup.ilm.pattern: "{now/d}-000001"

setup.template.enabled: true
setup.template.overwrite: false ## <-- After Setup set to false
setup.template.name: "metricbeateoprodsec-%{[agent.version]}" 
setup.template.pattern: "metricbeateoprodsec-%{[agent.version]}-*" 

setup.template.settings:
  index.number_of_shards: 1
  index.number_of_replicas: 1
  index.codec: best_compression

setup.dashboards.enabled: false ## <-- After Setup set to false
setup.dashboards.index: "metricbeateoprodsec-*"

```

Now using this updated config above Start metricbeat either

`metricbeat -e`

`systemctl start metricbeat`

Now the data is flowing into the correct index

 ![Screen Shot 2022-11-30 at 5.22.25 PM](https://us1.discourse-cdn.com/elastic/original/3X/9/d/9dcb9d47808a280b44f0ff56c69799d93ae7cf95.png)

And the Dashboards are working (I even looked inside them and they are using the correct index pattern)

 ![Screen Shot 2022-11-30 at 5.23.03 PM](https://us1.discourse-cdn.com/elastic/original/3X/a/f/af706f3b6b9d8485da27c5cede9cd64652820a4f.jpeg)

And the data is flowing into the correct index pattern and you can see that in Discover

 ![Screen Shot 2022-11-30 at 5.24.48 PM](https://us1.discourse-cdn.com/elastic/original/3X/e/9/e904d0177d6934f416b07554d78aa2833cdd5f7f.png)

---

<div class="post-metadata">

**Author:** ![Ankita\_Pachauri](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ankita_pachauri/32/61469_2.png) [@Ankita\_Pachauri](https://discuss.elastic.co/u/Ankita_Pachauri)\
**Post date:** [December 7, 2022, 5:57am UTC](https://discuss.elastic.co/t/index-lifecycle-management-metricbeat/318757/19 "2022-12-07T05:57:18Z")

</div>

Thanks team, it worked. However, deletion policy is automatically deleted from the ILM. Can you please let me know the same?

Can i also get support for metricbeat ILM policy for kubernetes installation and heartbeat as well?

//Ankita

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [December 7, 2022, 7:21am UTC](https://discuss.elastic.co/t/index-lifecycle-management-metricbeat/318757/20 "2022-12-07T07:21:21Z")

</div>

> [@Ankita\_Pachauri](#):
>
> However, deletion policy is automatically deleted from the ILM.

Just go into the UI and adjust the ILM policy to whatever you like.

> [@Ankita\_Pachauri](#):
>
> Can i also get support for metricbeat ILM policy for kubernetes installation and heartbeat as well?

ILM is available for all the beats out of the box.... Really, it would be my suggestion that you would just install them and use them with the default settings, learn how they all work and then adjust from there. If you use the defaults defaults and not worry about these individual index names You would probably make much more progress. You could just as easily put a tag with the source name and then filter on the dashboards by those tags.

Every beat has default ILM policies and you can just go into the UI and change those policies to whatever you like.

If you need details, you should open specific threads on each question on different topics.  
Perhaps stepping back and learning more about the overall concepts would help. Perhaps take some of the free training and there's lots of webinars and blogs on index lifecycle management.

[Next page](https://discuss.elastic.co/t/index-lifecycle-management-metricbeat/318757.md?page=2)