# Configuring ILM hot cold delete policy on ES Cluster **URL:** \ **Category:** Elasticsearch\ **Tags:** ilm-index-lifecycle-management\ **Created:** [December 23, 2020, 4:38pm UTC](https://discuss.elastic.co/t/configuring-ilm-hot-cold-delete-policy-on-es-cluster/259509 "2020-12-23T16:38:22Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![sandeepkanabar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sandeepkanabar/32/79399_2.png) [@sandeepkanabar](https://discuss.elastic.co/u/sandeepkanabar)\ **Post date:** [December 23, 2020, 4:38pm UTC](https://discuss.elastic.co/t/configuring-ilm-hot-cold-delete-policy-on-es-cluster/259509/1 "2020-12-23T16:38:22Z")
I'm thinking to apply ILM policies on my ES cluster containing historical data that needs to be preserved for legal purposes with retention period being 7 years (85 months). This cluster used to have day-wise indices. In order to boost search performances, I re-indexed day-wise indices into monthly indices. The customer is interested only in **last 18 months** data. Rest of the data is kept for legal purposes and is queried seldomly. Hence, I'm thinking to deploy hot- ~~warm~~ -cold-delete architecture in this. 1. The delete policy is straightforward --\> `Delete monthly indices older than 85 months.` 2. As per [ILM](https://www.elastic.co/guide/en/elasticsearch/reference/6.8/index-lifecycle-management.html), the current month index should be on hot node. Since my current month index is monthly which is just single index of around 500 GB primary with 12 shards and 1 replica, does it make sense to have a HOT node containing just 1 index or 2 hot nodes containing 1 index and its replica? I was thinking to go with `hot-cold-delete` architecture (no `warm`). i.e. keep `last 18 months data in hot nodes`. And rest of the data which is older than 18 months and less than 7 years in `cold nodes`. My questions: 1. Does this make sense? 2. Or you'd suggest to have just current month index and its replica on hot nodes and data from previous month till previous 18 months on warm nodes? 3. If I have to configure ILM, I've to specify `"min_age" : "31d"` in order for current monthly index to move from hot to warm/cold. Is that correct? Which means an index belonging to the month of April will not be moved to warm/cold node after 30 days but rather after 31 days. 4. For my use case, I don't need to configure `rollover`. ELK Stack Version is **6.8**. All past monthly indices are forcemerged and use best\_compression. Even the current month index uses best\_compression. The indexing penalty due to best\_compression on current index is acceptable since the data doesn't need to be immediately queried. Thanks ---
**Author:** ![sandeepkanabar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sandeepkanabar/32/79399_2.png) [@sandeepkanabar](https://discuss.elastic.co/u/sandeepkanabar)\ **Post date:** [December 24, 2020, 7:27am UTC](https://discuss.elastic.co/t/configuring-ilm-hot-cold-delete-policy-on-es-cluster/259509/2 "2020-12-24T07:27:22Z")
@Christian_Dahlqvist - any thoughts? will really appreciate some pointers here. Went through the excellently written [blog](https://www.elastic.co/blog/sizing-hot-warm-architectures-for-logging-and-metrics-in-the-elasticsearch-service-on-elastic-cloud). ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [December 26, 2020, 9:52am UTC](https://discuss.elastic.co/t/configuring-ilm-hot-cold-delete-policy-on-es-cluster/259509/3 "2020-12-26T09:52:16Z")
How many monthly indices do you have? What is the size of indices/shards for the monthly indices? How quickly would you need access to data older than 18 months? If it is for regulatory purposes, would restoring data from a snapshot be acceptable? Would you know which indices that would need to be restored? If you are planning to store large data volumes on cold nodes I would recommend upgrading to the latest 7.x version as it could [significantly reduce heap usage](https://www.elastic.co/blog/significantly-decrease-your-elasticsearch-heap-memory-usage). ---
**Author:** ![sandeepkanabar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sandeepkanabar/32/79399_2.png) [@sandeepkanabar](https://discuss.elastic.co/u/sandeepkanabar)\ **Post date:** [January 15, 2021, 6:25pm UTC](https://discuss.elastic.co/t/configuring-ilm-hot-cold-delete-policy-on-es-cluster/259509/4 "2021-01-15T18:25:34Z")
Hi Christian, Thanks for the prompt reply and apologies for the delayed response. Got pulled into other work and this took a backseat. > [@Christian\_Dahlqvist](#): > > How many monthly indices do you have? Currently, I have 85 monthly indices beginning from Jan-2014 till Jan-2021 (current month's index). Since retention period is 85 months, on 01-Feb-2021, the Jan-2014 monthly index would get purged. > [@Christian\_Dahlqvist](#): > > What is the size of indices/shards for the monthly indices? The monthly indices average about ~300 GB in size with 7 shards. Can change the shards to 5 if you'd suggest that or any other value. > [@Christian\_Dahlqvist](#): > > How quickly would you need access to data older than 18 months? If it is for regulatory purposes, would restoring data from a snapshot be acceptable? Would you know which indices that would need to be restored? Yes, restoring data from snapshot would be acceptable but it's better if data is directly available rather than triggering restore. We have a script to restore but NO, we do NOT know which indices out of those will contain the data. That's the problem. We will end up restoring all the indices older than 18 months which would defeat the purpose in first place. > [@Christian\_Dahlqvist](#): > > If you are planning to store large data volumes on cold nodes I would recommend upgrading to the latest 7.x version as it could [significantly reduce heap usage](https://www.elastic.co/blog/significantly-decrease-your-elasticsearch-heap-memory-usage). Thanks for this and also, my mappings are all optimised in fact this has just 4 mappings and the fields are all keywords and searches happen only on one field alone. The frozen tier will be perfect for my use case since I off-load all the data older than 18 months to frozen tier and can be searched using searchable snapshots but at this moment, frozen tier is not yet GA though I've been told it'll be GA soon. I also found that I cannot go for hot-cold because the [forcemerge](https://www.elastic.co/guide/en/elasticsearch/reference/6.8/_actions.html#ilm-forcemerge-action) action is only available for **WARM** phase. Thus, I'll need to go through HOT-WARM-COLD. Please correct me if my understand is wrong here. Your [blog](https://www.elastic.co/blog/sizing-hot-warm-architectures-for-logging-and-metrics-in-the-elasticsearch-service-on-elastic-cloud) post mentions that > For nodes tuned for long-term data storage, it often makes sense to let them work as dedicated data nodes and minimize any additional work they need to perform. **Directing all queries either to hot nodes or dedicated coordinating only nodes can help achieve this**. Can you please shed light on how can I achieve this? i.e how can I route the queries to hot/warm nodes? Here's some info on my current configuration: The 6.8.6 cluster has 3 dedicated master nodes and 7 data nodes. [I can reduce the data nodes from 7 to 4 because the `best_compression` resulted in reduction of disk space by nearly \*\*40%\*\*]. The master nodes are minimal - 2 cores and 7GB RAM. The data nodes are powerful Azure VMs with 16 cores and 55 GB RAM. The dataset is around ~24TB. Out of that the last 18 months dataset including replica is ~12TB. I've set replicas = 0 for data older than 18 months. [All that data is snapshotted and read-only so can restore in case of data loss or node down]. Dataset older than 18 months comes to be ~12TB without replica. I'm beginning wonder if I'm adding too much complexity by hot-warm-cold here. The aim is to reduce costs but I think a lot of that has been achieved using `best_compression`. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [January 15, 2021, 6:38pm UTC](https://discuss.elastic.co/t/configuring-ilm-hot-cold-delete-policy-on-es-cluster/259509/5 "2021-01-15T18:38:14Z")
> [@sandeepkanabar](#): > > The monthly indices average about ~300 GB in size with 7 shards. Can change the shards to 5 if you'd suggest that or any other value. That sounds fine the way it is. I see no need to change that. > [@sandeepkanabar](#): > > I also found that I cannot go for hot-cold because the [forcemerge](https://www.elastic.co/guide/en/elasticsearch/reference/6.8/_actions.html#ilm-forcemerge-action) action is only available for **WARM** phase. Thus, I'll need to go through HOT-WARM-COLD. Please correct me if my understand is wrong here. As far as I know the fact that you go through different phases does not necessarily mean that you need to relocate shards to new nodes at every transition. I suspect you should be able to keep the data on the hot nodes for both HOT and WARM phases and then move it when it turns COLD. Not sure if this is something that might have changed over time. > [@sandeepkanabar](#): > > Can you please shed light on how can I achieve this? i.e how can I route the queries to hot/warm nodes? Only give hot nodes to the clients querying Elasticsearch. ---
**Author:** ![sandeepkanabar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sandeepkanabar/32/79399_2.png) [@sandeepkanabar](https://discuss.elastic.co/u/sandeepkanabar)\ **Post date:** [January 16, 2021, 3:21am UTC](https://discuss.elastic.co/t/configuring-ilm-hot-cold-delete-policy-on-es-cluster/259509/6 "2021-01-16T03:21:56Z")
Thanks a ton Christian for the super prompt replies. > [@Christian\_Dahlqvist](#): > > Only give hot nodes to the clients querying Elasticsearch. Excellent. So the clients should connect to the pool of `hot nodes` only. Got it. What about Kibana? The `elasticsearch.hosts` parameter in Kibana.yml should also connect to `hot` nodes or to all nodes? > [@Christian\_Dahlqvist](#): > > I suspect you should be able to keep the data on the hot nodes for both HOT and WARM phases and then move it when it turns COLD This is great to hear. So will forcemerge action still kick in when the respective no of days have elapsed even though there's no warm phase? ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [February 13, 2021, 3:21am UTC](https://discuss.elastic.co/t/configuring-ilm-hot-cold-delete-policy-on-es-cluster/259509/7 "2021-02-13T03:21:58Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.