Hello,
As the title says, all my streams share a common ILM policy but somehow it is ignored.
All nodes are capable of hot/warm/cold. It is a recent setup and we don't have separate roles yet.
Any pointer to what could be happening here?
You need to share your policy, it is impossible to troubleshoot without seeing it.
Share a screenshot of the policy in Kibana and the raw json as well.
PUT _ilm/policy/deusto-ilm-policy
{
"policy": {
"phases": {
"hot": {
"min_age": "0ms",
"actions": {
"set_priority": {
"priority": 100
}
}
},
"warm": {
"min_age": "1d",
"actions": {
"set_priority": {
"priority": 50
}
}
},
"cold": {
"min_age": "2d",
"actions": {
"readonly": {},
"set_priority": {
"priority": 0
}
}
},
"delete": {
"min_age": "60d",
"actions": {
"delete": {
"delete_searchable_snapshot": true
}
}
}
}
}
}
From what you shared, I'm not sure there is any issue.
Your lifecycle policy does not have a rollover, it has a warm phase tha will trigger after one day of the creation of the index, and a cold phase that will trigger after two days of the creation of the index.
If any index is older than 2 days they will be in cold.
It is also not clear if you are using normal indices or data streams, can you share the result of running GET _cat/indices in Kibana Dev Tools?
I'm not sure how the Streams app works, so I'm not sure if this could lead to any issues, but if all indices have the same role, it does not make any sense to have ILM with data tiering, you should have dedicated warm and cold indices for this to make sense.
I see what may be happening. The datastreams are not rolling daily into different indexes and thus not following the ILM as I expect. Is that so?
How I make the datastreams behave like I expect?
Thank you very much!
PS:
if all indices have the same role, it does not make any sense to have ILM with data tiering, you should have dedicated warm and cold indices for this to make sense.
I believe, but I could be wrong, that a cold index consumes less resources than a hot index. The ILM policy right now is just for that, though maybe in the future we should dedicate nodes to cold data retention.
Leandro has it: your policy has no rollover action, so ILM is never asked to start a new backing index, and the phase ages are measured from index creation. That is why an index more than two days old is already in cold.
There is a second thing worth checking. The cold phase's readonly action will not act on a data stream's write index; it waits at that step instead. So if the write index itself has reached cold, it can sit there without progressing. GET <data-stream>/_ilm/explain will tell you: look at phase, action, step, and at lifecycle_date_millis next to index_creation_date_millis, which is the timestamp the phase ages are counted from.
Adding rollover to hot is the fix going forward. As a replacement hot section, keeping your other phases as they are:
"hot": {
"min_age": "0ms",
"actions": {
"rollover": { "max_age": "1d", "max_primary_shard_size": "50gb" },
"set_priority": { "priority": 100 }
}
}
Two things that policy edit alone will not do. Check that the index template matching these streams actually applies this policy, since only new backing indices pick it up. And an index already in cold does not go back to hot because the policy gained a hot action, so you will probably need one manual POST <data-stream>/_rollover to get a fresh write index that starts in hot. Worth looking at the explain output for the current write index before you do that.
On expectations once it is rolling: max_age: 1d is approximately daily rather than midnight aligned, since ILM checks on a poll interval, and warm at 1d, cold at 2d and delete at 60d are then counted from each rollover. ILM also skips rolling an empty index by default, so you do not need min_docs. I would leave min_primary_shard_size out unless small indices actually become a problem, because a minimum size condition can hold rollover past max_age and that is the opposite of what you are after.
On cold being lighter than hot, which is the other half of what Leandro raised: this policy does not make a cold index smaller, and in your topology it does not put it on different hardware either. ILM injects a migrate action into warm and cold that sets index.routing.allocation.include._tier_preference, with cold resolving to data_cold,data_warm,data_hot. Your nodes carry all of those roles, and the generic data role as well, which takes precedence over the specialised ones. So the preference resolves to the same nodes whatever the phase says. Separate warm and cold nodes are what makes tiering mean something, which is the point he was making.
Disclosure: I'm a co-founder of log10x, which builds tooling to reduce data before it is indexed in Elasticsearch.
Dor
Thank you very much for your detailed answer!
I was under the impression that the ILM alone would make the data stream roll over a new index when a new phase started. Instead, the rollover conditions allow finer control as I understand it.
If the warm and cold phases only affect the index priority, then the texts "The warm tier is optimized for search performance over indexing performance." and "The cold tier is optimized for cost savings over search performance." would be some BS. I don't expect the indices to be smaller in disk size, unless using the Replicas, Shrink, Force Merge or Downsample options, but I thought that phasing those indices into warm or cold would mean less RAM usage for keeping them open.
Again, thank you very much. I hope your answers help other too!
Both descriptions for the warm and cold tier are correct, but it is expected that both warm and cold tiers have different hardware profiles from the hot tier.
If your nodes have all tiers then you do not have real data tiering and this will make no difference.