[ILM Policy] Best compromise between size and max age

Elastic Stack Elasticsearch
1

Hello Elasticians !

While testing ILM policy feature, I've just figured that the delete phase is relative to the rollover time, not the index creation time (source)

Let's say I have the following ILM policy :

{
"ims_customer_policy" : {
"version" : 1,
"modified_date" : "2019-10-02T09:59:43.574Z",
"policy" : {
"phases" : {
"hot" : {
"min_age" : "0ms",
"actions" : {
"rollover" : {
"max_size" : "30gb"
}
}
},
"delete" : {
"min_age" : "365d",
"actions" : {
"delete" : { }
}
}
}
}
}
}

This way, all index have a regular size (which is a good practice I think) but I can't be sure that logs will be conserved one year because an index could take one hour, or three days or whatever to reach the size of 30 GB right ? Some could be conserved some month while other could be conserved more than one year

My only option would be to add max age : 1 to my hot phase, but index size will not be regular (i.e more logs on week and less in weekends)

So, what is your best strategy to have a good compromise between retention time and index size ?

Thanks for your feedback !

2

Anyone ? @Christian_Dahlqvist @DavidTurner would you have some feedback on this ? :wink:

3

I would recommend setting a max age for the hot phase of perhaps a week as this seems reasonable given your retention period. If this results in some indices that are smaller than the size target that should not be a problem.

Then adjust you retention settings so you always keep at least a years worth of data. If the oldest index covers a full week this means you at times will hold a year plus a week which in my experience generally is fine.

4

Hello @Christian_Dahlqvist. Mmh. I just have hot nodes as I don't have enough resources to have warm and cold nodes :expressionless:

5

That does not matter. Moving shards between zones is one aspect of the index lifecycle, but there are other steps that apply even if you only have a single zone.

6

Ok thank you @Christian_Dahlqvist. I'm gonna try this and see what is going on !