Hello nice elastic people.
I am currently toying with ILM (7.x, not yet upgraded to 8.0) and ran into a surprising behavior.
I have configured ILM to increase the number of replicas when transitioning from hot to cold.
However, at least when triggering the ILM transition manually by Move to lifecycle step API | Elasticsearch Guide [8.0] | Elastic, it seems to behave unexpectedly.
I start with an index on the data_hot nodes (replicas: 0).
Then I manually trigger ILM transition.
ILM configures the desired number of replicas (1).
ES starts replicating the index onto another data_hot node.
ILM waits for the replication to finish.
ILM configures _tier_preference and indices start to migrate to data_cold(s).
Why does it work like that?
Why is the replica created on the hot nodes just to be migrated away immediately?
What did I miss?
What is preventing this new replica from being put directly onto data_cold(s)?