Elastic ILM

Elastic Stack Elasticsearch
1

Hi all,

I have a use case where my index should remain in the hot phase for 3 days, then move to the cold phase for 11 days, followed by the frozen phase for 90 days, after which the data should be deleted.

My concern is regarding the index rollover setting. The default rollover period is 30 days, but since my ILM policy moves data to the cold phase in just 3 days, should I adjust the rollover period to 3 days as well?

Would appreciate any insights on this.

Thank you!

2

yes, you can adjust the period at which the rollover occurs, this will allow you to better control the time at which you want to move data between the different tiers.

For your case, I would say to make sure to rollover every day or 50GB (recommended limit for 1 primary shard).

Be careful with small shards though.

you can test some things like this :

PUT _ilm/policy/<policyName>
{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": {
            "max_age": "1d",
            "max_primary_shard_size": "50gb"
          },
          "set_priority": {
            "priority": 100
          }
        },
        "min_age": "0ms"
      },
      "cold": {
        "min_age": "3d",
        "actions": {
          "set_priority": {
            "priority": 0
          }
        }
      },
      "frozen": {
        "min_age": "11d",
        "actions": {
          "searchable_snapshot": {
            "snapshot_repository": "found-snapshots"
          }
        }
      },
      "delete": {
        "min_age": "90d",
        "actions": {
          "delete": {
            "delete_searchable_snapshot": true
          }
        }
      }
    }
  }
}

It all depends on the rollover date.
feel free to adjust this example and add additional features if needed

3

The time in the hot zone starts counting when rollover occurs so if your rollover period is 30 days you may have up to 33 days of data in the hot zone. I would set it to 3 days if you are OK having between 3 and 6 days worth of data in the hot zone at any time. Do not go too low as that will result in a lot of small shards.