Elastic ILM Filebeat

[lehu]

Hi,

so I created a working lifecycle policy and I want it to apply it to the indices that are daily created by filebeat. But my filebeat configuration uses setup.template.name and setup.template.pattern and I dont want to change it to avoid the tons of mappings. But I cant define a policy for filebeat because If index lifecycle management is enabled (which is typically the default), setup.template.name and setup.template.pattern are ignored.. Is there any way to solve this problem?

[stephenb (Stephen Brown)]

Hi @lehu

What Version? Very Important.

Share your filebeat.yml.

You should be able to add the ILM policy to the template and then use the write alias as the output in filebeat.

Set the ILM policy to rollover daily, and you should be good.

[lehu]

Version: 8.10.4

This is basically how my filebeat.yml looks like:

filebeat.inputs:

- type: filestream

  id: my-filestream-id

  enabled: true

  paths:
    - my_path

  parsers:
    - ndjson:
        target: ""
        overwrite_keys: true
        add_error_key: true

setup.template.settings:
  index.number_of_shards: 1
  
setup.template:
  name: "test"
  pattern: "test-*"
  enabled: false

cloud.id: my_cloud_id

output.elasticsearch:
  api_key: my_api_key
 
  index: "test-%{[agent.version]}-%{+yyyy.MM.dd}"

logging.level: debug
logging.to_files: true
logging.files:
  path: /var/log/filebeat
  name: filebeat
  keepfiles: 7
  permissions: 0640

Thanks for the answer

[stephenb (Stephen Brown)]

Take a look at this.

Change the index name | Filebeat Reference [8.10] | Elastic

This shows the configuration.

Run setup etc.

If you want daily backing indices, then you will set the rollover to 1d in the ILM policy, not on the index name i.e. don't use -%{+yyyy.MM.dd} ... In General, you do not use the Daily Naming and ILM together... its one or the other... with ILM rollover set to 1 day you effectively get Daily indices... that will look like

.ds-filebeat-8.10.4-2023.10.15-000001

[lehu]

Thank you very much!