How do I setup different retentions

Elastic Stack Elasticsearch
1

We are currently using Curator to delete specific indicies at 30, 60 and 90 days. We have added 2 warm and 2 cold nodes to our cluster and still want to be able to do the same retention with ILM. Right now we just have a single index template, so I am struggling to find the best way to do so.

Should I break out the single index template into individual templates for each index/log source? Or is there a way to use ILM in the same way we use Curator where indexes that start with XXX-* are deleted at 30 days, YYY-* are deleted at 90?

Any suggestions on how I should go about reconfiguring things?

1 Like
2

You will need an ILM policy for each source, and that will mean a template for each one.

3

What about applying multiple index templates when creating an index? Doing so seems like it would allow me to achieve three things:

  1. The ability to set the ILM retention policies for 30, 60, 90 days depending on that indexes requirements
  2. The ability to set index settings, like the field mapping, depending on that indexes requirements
  3. A fallback, catch all index template(s) and ILM policy that will set our default shard to replica count and delete indexes after 30 days if that index is not added to a template that has longer retention

Lets say I have two ILM policies:

  1. Hot-Warm-Delete30_Policy
    a. Moves from hot to warm nodes
    b. Deletes after 30 days
  2. Hot-Warm-Cold-Delete90_Policy
    a. Moves from hot to warm to cold nodes
    b. Deletes after 90 days

Lets say I have three templates:

  1. ILM_90_day_delete_template
    a. Index pattern: firewalls-* (Would contain multiple patterns)
    b. Priority: 20
    c. ILM Policy: Hot-Warm-Cold-Delete90_Policy
  2. Firewall_template
    a. Index pattern: firewalls-*
    b. Priority: 10
    c. Sets a field mapping
  3. ILM_30_day_delete_template
    a. Index pattern: *
    b. Priority: 0
    c. ILM Policy: Hot-Warm-Delete30_Policy
  4. Default_template
    a. Index pattern: *
    b. Priority: 0
    c. Sets shard and replica count to 2:1

If an index called ā€œapplication-log-11162021ā€ was created it would get the ā€˜ILM_30_day_delete_templateā€™ and the ā€˜Default_templateā€™ to set the shard/replica count and delete after 30 days.

Now when an index called ā€œfirewalls-log-11162021ā€ was created it would get a combination of all four templates. But the ā€˜ILM_90_day_delete_templateā€™ would take priority over the ā€˜30_day_delete_templateā€™ as it has a higher priority. As well as the field mapping and the shard/replica count template.

Does this sound correct? Is there a better way or best practices of achieving the requirements? Any suggestions would be greatly appreciated.

4

That all sounds good, I would definitely try it to make sure it works as expected.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.