ES custom ILM policy with cumulative index or disk size

Elastic Stack Elasticsearch
1

We're trying to implement an ILM policy for moving indices between hot -> warm -> cold in out on-premise server.

What we have is this:

  1. SSD for storing hot indices
  2. HDD for storing warm indices > 7d
  3. NAS for storing cold indices > 30d

Considering that we will not be able to increase the size of the disks on demand, we want to make sure we get the ILM policies to self police in such a way that -

HOT TIER:

  1. Hot index moves to warm if it's age is > 7d.
  2. Oldest hot index moves to warm, if the total cumulative size of all hot indices is > 10TB

WARM TIER

  1. Warm index moves to cold if it's age is > 30d
  2. Oldest warm index moves to cold, if the total cumulative size of all warm indices is > 50TB

We are very specific of point 2 in both the cases in order to manage the SSD and HDD better since it cannot be expanded on a whim.

Is it possible to implement an ILM policy with the above parameters and conditions? If so, where would I begin?

2

You can't do it on cumulative, no. Policies only apply to indices (and their shards), not to tier usage beyond time-based limits.

3

Thanks for the quick response. So is there no other way to self police the storage / migration? Does this solely fall upon the infra management?

4

To do what you want to do, yes that is correct.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.