I am being forced to build an ES cluster which will have new indexes created every few days without ever deleting any indices or docs (infinite retention).
I’m going to leverage ILM so I can reduce long-term costs of managing this cluster. I’ll be reducing indices to 1 shard and force merging them to a single segment during the “Warm” phase, yet all the ILM docs direct to use the Deleted phase to eventually delete old indices, which this cluster will not do .
My current belief is that, given the endless-retention constraint, ILM is going to be the best bet for reducing cost and ensuring good query efficiency. I’ll add data nodes to each phase as the data grows over the next decade.
Is my assumption incorrect? Am I missing something?