I do logfile analysis trainings which include the Elastic Stack.
I set setup.ilm.policy_name: metricbeat in /etc/metricbeat/metricbeat.yml in order to have a single ILM policy for different Metricbeat versions. Otherwise I'd have a new ILM policy with standard 50 GiB setting for each (even minor) release of the beat. I simply do not get why I would like to set up a different policy for each version. I know I could tell it to load my own policy, but I'd rather just change it in Kibana, rollover manually if needed, and be done with it.
This partly works:
- The Beat creates the new ILM policy.
- It creates a policy with the name.
- It creates a new index (I removed the old one, training demo environment)
- But it applies the wrong policy which leads to
illegal_argument_exception: policy [metricbeat-7.1.1] does not existerror message in Kibana
So I used:
setup.template.settings.index.lifecycle.name: "metricbeat"
setup.ilm.policy_name: "metricbeat"
but it still gives the same error in Kibana, despite Metricbeat having overrode the index template as I told it to:
2019-05-29T15:31:17.736+0200 INFO template/load.go:84 Existing template will be overwritten, as overwrite is enabled.
Is that a bug?
What is the reasoning behind cluttering Elasticsearch with one ILM policy per beat version? Is this complexity really required?