Is there a limit on the number of ILM Policies that can be created in a Cluster?

Elastic Stack Elasticsearch
1

I have a situation where I need to create 120 indices , wherein each index has the data for a particular month.

  • The data is stored for past 10 years (12 month*10 year = 120 indices).
  • Each index hold 1 month Data.
  • I have a requirement wherein I need to delete the oldest 1 month data every month. Since , these are old data, there is no write operation in these indexes.
  • Question : Can I create ILM Policies for each Index , as to when the Index completes the assigned time within the WARM phase, it moves to the DELETE phase and the Index is deleted? (Example: If an Index hold 6 years old data for Jan month, then this Index should get deleted after it completes 4 years and likewise)
  • So, I will be needing around 120 ILM policies(as each Index will have unique deletion min_age) , each policy attached to an Index and Once the Index is Deleted, we will delete the corresponding Policy .
  • So, Is creation of 120 ILM Policies recommended by Elastic? and Will it effect the overall performance ?
2

I don't think there is a limit on the number of ILM policies, you can have multiple ILM policies and 120 will work without any issue.

But I don't think you need one ILM policy for each index, if you need to delete the oldest 1 month data, just one policy should be enough.

For example, if you will have 10 years of data and monthly indices, you will have something like this:

indexname-2013-01
indexname-2013-02
indexname-2013-03
...
...
indexname-2023-01

Then you would have a ILM policy with a DELETE phase that would trigger after 10 years, so in February 2023 the index indexname-2013-01 should be deleted, in March 2023, the indexname-2013-02 should be deleted and it goes on, one policy would be enough.

The main issue here is that ILM uses the creation date of the index to control the phases, so you will need to add a setting named index.lifecycle.origination_date in every one of your indices with the corresponding date for the index in Unix epoch time, for example, the value of this setting for the indexname-2013-01 would be the correspondent epoch time in miliseconds of January 1st, 2013, you can read more about this setting in the documentation, also read this blog post about timing the phases.

4 Likes
3

Thanks a lot !! you are a lifesaver . This is perfectly what we need . !!
We will be trying out the solution .

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.