Applying ILM on custom index

Elastic Stack Elasticsearch
1

Hi,

I am using filebeat 8.3.3 with several inputs and writing them to the same ES 8.3.3. To separate the different inputs on ES, I have the following in my filebeat.yml.

output.elasticsearch:
  indices:
    - index: "filebeat-%{[agent.version]}-inputA-%{+yyyy.MM.dd}"
      when.equals:
        input.type: "log"
    - index: "filebeat-%{[agent.version]}-inputB-%{+yyyy.MM.dd}"
      when.equals:
        input.type: "filestream"

I now have these indices created daily, but ILM is not applied on them, so they can grow to 1-2TB per day.

I've read that ILM doesn't work on custom indices. Is there any workaround to this so that I can apply ILM on these 2 indices and have them be rotated every 50GB and deleted after 7 days?

Thank you.

2

Hi @hjazz6,

ILM supports custom indices but you need to use an alias and often you have to bootstrap the index. See here for the following documentation.

3

Also FWIW there's no value in using %{+yyyy.MM.dd} in your index names if you are using ILM as variables won't be used when things are rolled over.

4

Thanks, I've removed the date from the index names.