Index Lifecycle Management configuration file setup

Elastic Stack Elasticsearch
1

Hi,
Currently have a set up on linux host, using 6.7.1 in the stack - elasticsearch, logstash, kibana, filebeat, metricbeat, etc.

I'm trying to set this up for the data we currently have, and to management indexes yet to be created

So the documentation mostly shows creating the ILM config through the API, but I'm wondering if there is documentation around strictly setting this up through the configuration files, eg. elasticsearch.yml, filebeat.yml,etc.

Has anyone set it up like this? I'd rather not just create policies through kibana UI as I'd like to set up difference environments, and config files for ILM would make it easier.

Would a policy be created under the elasticsearch directory?
then index templates, for example, under filebeat, and then reference the template under output.elasticsearch in some way? I'm under the assumption an alias would be needed for ILM, so this would need to be done under an index template, as all our filebeat indexes are created based on the date

2

Hi Karl,

Currently the API is the only way to configure ILM on the Elasticsearch side. You can configure the beats portion of the configuration in filebeat.yml, but for ES you cannot put a policy on disk for configuration.

3

oh okay, is there any reason it's not possible? I figured it'd be the best way to keep any environments in sync without the manual creation
So if I make the changes to my beats for the policies, then deploy them into a prod environment, I guess I would need to create the policy manually first, I suppose that'd be 100% fine due to the policy not being assigned yet to anything.

For old indexes that have no ILM - should I create a seperate policy to move them from hot -> warm -> cold (with no rollover) and then assign them in the ui? or would they need a template also?

4

Having disk-based configuration for policies (and templates) allows an issue where one node has a different file than another and there is conflict in loading the configuration.

You shouldn't need a template for these. Templates are only for newly created indices. In your case you could either create a new policy that didn't have the rollover in it, or you can set the index.lifecycle.indexing_complete setting to skip the rollover for an existing policy. You should then be able to use your policy without creating a new index (it'll skip rollover and move to the next step/phase).

5

Oh okay fair enough, I can create the policy first on elasticsearch before deploying changes for beats configuration.
With the beats configuration, it doesn't seem to create the alias - so I'm not sure if I've set it up right, or maybe I'm missing something:

setup.ilm.enabled: auto
setup.template.enabled: true
setup.template.name: "metricbeat-6.7.1"
setup.template.pattern: "metricbeat-*"
setup.template.settings:
  index.number_of_shards: 1
  index.codec: best_compression
  index.lifecycle.name: "metricbeat"
  index.lifecycle.rollover_alias: "metricbeat-6.7.1"
setup.template.overwrite: true

this is just an example of the config i have for metricbeat
my understanding is that metricbeat will create the index daily, and it should also create the alias assigned to all metricbeat-* indices?

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.