I have a hot-warm architecture cluster running in production containing 1000+ daily indices spanning 3 months. A single ILM policy moves indices from hot to warm after 4 days of creation, then to cold phase after 30 days and delete phase after 90 days.
Currently all of these indices have this ILM policy applied directly via templates.
Warm phase is configured to move an index from hot nodes to warm nodes. Problem here is that warm nodes are running out of disk space where as hot nodes have ample space left.
Hence I attempted to update the warm phase timing from 4 days to say, 6 days to increase retention on hot nodes. But the cluster doesn't seem to accept this change either via Kibana or via REST API. This could be because there are indices currently executing this phase.
What are my options here apart from deleting old indices or adding more disk?
Any help would be appreciated, thanks!