Assigning ILM for metricbeat

[elasticforme (Sachin)]

I have started metricbeat used custome field.yml file to load template.
I have following in metricbeat.yml

setup.template.fields: "fields.yml"
setup.ilm.enabled: auto
setup.template.overwrite: true

now I started metricbeat on remote system and send data to logstash, it did created index
metricbeat-7.9.1 but didn't had any ILM assign. I am trying to assign ILM to that index and gives me this error. what am I doing wrong?

[ChrsMark (Chris Mark)]

Hi!

From Configure index lifecycle management | Metricbeat Reference [8.11] | Elastic

When auto (the default) is specified on version 7.0 and later, Metricbeat automatically uses index lifecycle management if the feature is enabled in Elasticsearch and has the required license; otherwise, Metricbeat creates daily indices.

Can you check if it could be a licence issue or if your Elasticsearch is properly configured?

[elasticforme (Sachin)]

yes I do have setup.ilm.enable:auto

this is running basic license.

I delete everything, recreated everything but still same. Data is coming from logstash and has following. It does create this index properly when I start logstash "metricbeat-7.9.1"

output {
elasticsearch {
hosts => ["elktst02:9200","elktst03:9200"]
index => "%{[@metadata][beat]}-%{[@metadata][version]}"
user => "elastic"
password => "changeme"
}
stdout { codec => json_lines }
}

and this is metricbeat.yml

setup.template.fields: "fields.yml"
setup.ilm.enabled: auto
setup.ilm.rollover_alias: "metricbeat-%{[agent.version]}"
setup.ilm.pattern: "{now/d}-000001"
setup.template.overwrite: true
setup.template.settings:
index.number_of_shards: 1
index.codec: best_compression

[ChrsMark (Chris Mark)]

Did you run metricbeat setup? https://www.elastic.co/guide/en/beats/metricbeat/current/metricbeat-template.html#load-template-manually

This will be needed in order to properly setup the indexes and ilms defined in Metricbeat config.

[elasticforme (Sachin)]

I did went through that whole document many time over. Need help telling me exactly what I need. getting very confused. what I need is
metricbeat name = metricbeat-%{[agent.version]}

and then when it rolls over it should be just metricbeat-%{[agent.version]}-yyyy.dd-00001

this is what I have in metricbeat.yml file

setup.template.fields: "fields.yml"
setup.ilm.enabled: auto
setup.ilm.rollover_alias: "metricbeat-%{[agent.version]}"
setup.ilm.pattern: "{+yyyy.MM}-000001"
setup.template.overwrite: true
setup.ilm.overwrite: true

and then from logstash I am creating index = metricbeat-7.9.3

[ChrsMark (Chris Mark)]

Hi!

Running metricbeat setup --index-management -E output.logstash.enabled=false -E 'output.elasticsearch.hosts=["localhost:9200"]' with the proper hosts defined will do the setup.
I would suggest trying this incrementally so as to figure out what exactly configuration you need.
For example I would try with:

setup.template.name: "your_template_name"
setup.template.fields: "path/to/fields.yml"
setup.template.overwrite: true
setup.template.pattern: "some-*"

Then if you verify that it works you can add more settings like ilm etc.

C.