# Does single node Elasticsearch supports ILM ploicy **URL:** \ **Category:** Kibana\ **Created:** [December 29, 2023, 11:46am UTC](https://discuss.elastic.co/t/does-single-node-elasticsearch-supports-ilm-ploicy/350124 "2023-12-29T11:46:20Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![Ravi\_Pattar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ravi_pattar/32/124209_2.png) [@Ravi\_Pattar](https://discuss.elastic.co/u/Ravi_Pattar)\ **Post date:** [December 29, 2023, 11:46am UTC](https://discuss.elastic.co/t/does-single-node-elasticsearch-supports-ilm-ploicy/350124/1 "2023-12-29T11:46:20Z")
Hello, I have applied the ILM policy because the disk space usage was seen very high. But I don't see any changes in the disk space after implementing the ILM policy for filebeat. ```auto shards disk.indices disk.used disk.total disk.percent 100 111 199.1gb 205.4gb 242gb 84 ``` ```auto 207G / 202G /var 201G /var/lib 200G /var/lib/elasticsearch/nodes/0/indices 200G /var/lib/elasticsearch/nodes/0 200G /var/lib/elasticsearch/nodes 200G /var/lib/elasticsearch 15G /var/lib/elasticsearch/nodes/0/indices/dE43jALIRbmNM5jBMj8ZDw/0/index 15G /var/lib/elasticsearch/nodes/0/indices/dE43jALIRbmNM5jBMj8ZDw/0 15G /var/lib/elasticsearch/nodes/0/indices/dE43jALIRbmNM5jBMj8ZDw ``` Thanks, ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [December 29, 2023, 1:47pm UTC](https://discuss.elastic.co/t/does-single-node-elasticsearch-supports-ilm-ploicy/350124/2 "2023-12-29T13:47:45Z")
> [@Ravi\_Pattar](#): > > I have applied the ILM policy because the disk space usage was seen very high. What does your policy looks like? You need a delete phase so your data would be deleted after some time to free up space. ---
**Author:** ![Ravi\_Pattar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ravi_pattar/32/124209_2.png) [@Ravi\_Pattar](https://discuss.elastic.co/u/Ravi_Pattar)\ **Post date:** [December 29, 2023, 2:18pm UTC](https://discuss.elastic.co/t/does-single-node-elasticsearch-supports-ilm-ploicy/350124/3 "2023-12-29T14:18:25Z")
> [@leandrojmp](#): > > What does your policy looks like? Hot phase (Required) Rollover: enabled Maximum primary shard size: 50G Maximum age: 7 days Warm phase Move data into phase when:5 days old Cold phase Move data into phase when:10 days old Delete phase Move data into phase when:25 days old ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [December 29, 2023, 2:22pm UTC](https://discuss.elastic.co/t/does-single-node-elasticsearch-supports-ilm-ploicy/350124/4 "2023-12-29T14:22:32Z")
You have just one node, you do not need a warm and cold phase as this would do nothing, you can remove those steps. Also, assuming that your data will rollover after 7 days, it may take 32 days for it to be deleted, you need to check if this time range is enough. From what you shared it seems that you have a lot of data for a small cluste and may need to reduce both the time that it takes to delete and rollover, and the shard size. ---
**Author:** ![Ravi\_Pattar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ravi_pattar/32/124209_2.png) [@Ravi\_Pattar](https://discuss.elastic.co/u/Ravi_Pattar)\ **Post date:** [December 29, 2023, 2:50pm UTC](https://discuss.elastic.co/t/does-single-node-elasticsearch-supports-ilm-ploicy/350124/5 "2023-12-29T14:50:54Z")
> [@leandrojmp](#): > > You have just one node, you do not need a warm and cold phase as this would do nothing, you can remove those steps. > > Also, assuming that your data will rollover after 7 days, it may take 32 days for it to be deleted, you need to check if this time range is enough. > > From what you shared it seems that you have a lot of data for a small cluste and may need to reduce both the time that it takes to delete and rollover, and the shard size. Hello Leandro, Thanks for the response. Please confirm if I can go with the below new changes as this is a production server. Hot phase (Required) Rollover: enabled Maximum primary shard size: 10G Maximum index size: 10G Maximum age: 7 days Delete phase Move data into phase when:15 days old Also do we need to reboot the server after the changes? ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [December 29, 2023, 3:34pm UTC](https://discuss.elastic.co/t/does-single-node-elasticsearch-supports-ilm-ploicy/350124/6 "2023-12-29T15:34:55Z")
> [@Ravi\_Pattar](#): > > Hot phase (Required) > Rollover: enabled > Maximum primary shard size: 10G > Maximum index size: 10G > Maximum age: 7 days > > Delete phase > Move data into phase when:15 days old Yes, that seems reasonable. But keep in mind that the following phase and your case the delete phase is calculated from the time of rollover which you have set as a maximum of 7 days in the hot. So index take 7 days to roll over plus 15 days until delete. You could have data that's 22 days old and then it will delete the 7 days worth of data since the index contain 7 days where the data. If you're indices, fill up faster. Rollover faster. The same math implies just with a shorter duration But yes, that's all seems reasonable > [@Ravi\_Pattar](#): > > Also do we need to reboot the server after the changes? No, you do not need to reboot the system. You may need to force a rollover on the existing indices depending on what version of Elasticsearch and the ILM phase they're in, in order to pick up/ apply the new ILM policy. ---
**Author:** ![Ravi\_Pattar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ravi_pattar/32/124209_2.png) [@Ravi\_Pattar](https://discuss.elastic.co/u/Ravi_Pattar)\ **Post date:** [December 29, 2023, 3:52pm UTC](https://discuss.elastic.co/t/does-single-node-elasticsearch-supports-ilm-ploicy/350124/7 "2023-12-29T15:52:25Z")
> [@stephenb](#): > > So index take 7 days to roll over plus 15 days until delete. You could have data that's 22 days old and then it will delete the 7 days worth of data since the index contain 7 days where the data. Thanks, Stephen. What I understand from the above is I can have the data available for last 22 days and once the delete operation occurs I hope the latest 7 days data would be available. Also the ELK version here is 7.17.15 ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [December 29, 2023, 4:09pm UTC](https://discuss.elastic.co/t/does-single-node-elasticsearch-supports-ilm-ploicy/350124/8 "2023-12-29T16:09:55Z")
> [@Ravi\_Pattar](#): > > What I understand from the above is I can have the data available for last 22 days and once the delete operation occurs I hope the latest 7 days data would be available. Let say your indices look like this, rolling over every 7 days ```auto my-index-000001 Jan 1 - Jan 7 my-index-000002 Jan 8 - Jan 15 my-index-000003 Jan 16 - Jan 23 my-index-000004 Jan 24 - Feb 1 .... ``` On Jan ~22nd (15days after rollover on Jan 7) my-index-000001 with data Jan 1 - Jan 7th would be deleted The other indices would still be there. The best way to do this is to get started, observe, and adjust. ---
**Author:** ![Ravi\_Pattar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ravi_pattar/32/124209_2.png) [@Ravi\_Pattar](https://discuss.elastic.co/u/Ravi_Pattar)\ **Post date:** [December 29, 2023, 5:57pm UTC](https://discuss.elastic.co/t/does-single-node-elasticsearch-supports-ilm-ploicy/350124/9 "2023-12-29T17:57:50Z")
Thank you Stephen for giving more clarity. ---
**Author:** ![Ravi\_Pattar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ravi_pattar/32/124209_2.png) [@Ravi\_Pattar](https://discuss.elastic.co/u/Ravi_Pattar)\ **Post date:** [January 4, 2024, 7:49am UTC](https://discuss.elastic.co/t/does-single-node-elasticsearch-supports-ilm-ploicy/350124/10 "2024-01-04T07:49:50Z")
Hello, As discussed in the previous post that ILM policy for filebeat was created on 29th Dec-23 as below. Hot phase (Required) Rollover: enabled Maximum primary shard size: 10G Maximum index size: 10G Maximum age: 7 days Now, I am seeing the below message on the kibana GUI which is a production server. ```auto You are editing an existing policy. Any changes you make will affect 32 linked indices(opens in a new tab or window) and **1 linked index template** that are attached to this policy. Alternatively, you can save these changes in a new policy. And 1 linked index template it shows as filebeat-7.17.15. ``` Question: Is anything wrong here? Please suggest. I am worried if the 1 linked template as mentioned also gets deleted and create a system wide issue. Thanks, ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [January 4, 2024, 5:32pm UTC](https://discuss.elastic.co/t/does-single-node-elasticsearch-supports-ilm-ploicy/350124/11 "2024-01-04T17:32:21Z")
> [@Ravi\_Pattar](#): > > Now, I am seeing the below message on the kibana GUI which is a production server. > > ```auto > You are editing an existing policy. Any changes you make will affect 32 linked indices(opens in a new tab or window) and **1 linked index template** that are attached to this policy. Alternatively, you can save these changes in a new policy. And 1 linked index template it shows as filebeat-7.17.15. > > ``` > > Question: Is anything wrong here? Please suggest. I am worried if the 1 linked template as mentioned also gets deleted and create a system wide issue. If you are seeing this? ![Screenshot 2024-01-04 at 9.29.42 AM](https://us1.discourse-cdn.com/elastic/original/3X/2/5/25ae0511d4892bb881dd840489b0c73ed49a1cf4.png) No that is just a standard awareness warning. The policy change can/will affect the already linked indices. The template will not be deleted or changed in any way. New Indices created by the template will have this new policy applied. I do want to be clear this is _ **your** _ production system ... if you are concerned you should test in a non-production system. ---
**Author:** ![Ravi\_Pattar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ravi_pattar/32/124209_2.png) [@Ravi\_Pattar](https://discuss.elastic.co/u/Ravi_Pattar)\ **Post date:** [January 5, 2024, 5:23am UTC](https://discuss.elastic.co/t/does-single-node-elasticsearch-supports-ilm-ploicy/350124/12 "2024-01-05T05:23:53Z")
> [@stephenb](#): > > No that is just a standard awareness warning. > The policy change can/will affect the already linked indices. > The template will not be deleted or changed in any way. > New Indices created by the template will have this new policy applied. Thanks, Stephen ---
**Author:** ![Ravi\_Pattar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ravi_pattar/32/124209_2.png) [@Ravi\_Pattar](https://discuss.elastic.co/u/Ravi_Pattar)\ **Post date:** [January 5, 2024, 5:28am UTC](https://discuss.elastic.co/t/does-single-node-elasticsearch-supports-ilm-ploicy/350124/13 "2024-01-05T05:28:16Z")
> [@stephenb](#): > > I do want to be clear this is _ **your** _ production system ... if you are concerned you should test in a non-production system. Yeah, this is on the production server. Also I have implemented the same ILM policy for filbeat in test instance before applying the ILM in production server. Below is the ss from my test instance where it is not showing anything for **1 linked index template** ```auto # Edit policy filebeat **You are editing an existing policy.** Any changes you make will affect [4 linked indices that are attached to this policy. Alternatively, you can save these changes in a new policy. Save as new policy ``` ---
**Author:** ![Ravi\_Pattar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ravi_pattar/32/124209_2.png) [@Ravi\_Pattar](https://discuss.elastic.co/u/Ravi_Pattar)\ **Post date:** [January 6, 2024, 6:16pm UTC](https://discuss.elastic.co/t/does-single-node-elasticsearch-supports-ilm-ploicy/350124/14 "2024-01-06T18:16:57Z")
> [@stephenb](#): > > Yes, that seems reasonable. But keep in mind that the following phase and your case the delete phase is calculated from the time of rollover which you have set as a maximum of 7 days in the hot. > > So index take 7 days to roll over plus 15 days until delete. You could have data that's 22 days old and then it will delete the 7 days worth of data since the index contain 7 days where the data Hi @stephenb Per our earlier discussions on this thread. The ILM policy was set for filebeat index as below on 29th Dec-23. ```auto Hot phase (Required) Rollover: enabled Maximum primary shard size: 10G Maximum index size: 10G Maximum age: 7 days Delete phase Move data into phase when:15 days old ``` Note: I was expecting the last 7 days data should have been deleted by now but that's not the case because I could see no changes. ![image](https://us1.discourse-cdn.com/elastic/original/3X/a/f/af4993df9469ffb25f4aa50852290ede3d823ed1.png) Below details on the ILM set. ![image](https://us1.discourse-cdn.com/elastic/original/3X/f/6/f65164567dde1f9effc25f09034c11796615817a.png) ![image](https://us1.discourse-cdn.com/elastic/original/3X/1/e/1ec64dff610572625f53243e1336c95709549ea9.png) Correct me if I am wrong on my understanding here. Is it going to take 22 days to run the delete operation? when the max age was set to 7 days in hot phase and 15 days to move data into delete phase. If this is a case then can I change it to lesser value other than 15 days e.g 5 or \< 10 days etc in the delete phase instead of waiting for another 15 days more because I can see that the disk space utilization has started piling up. GET \_cat/allocation?v&s=disk.indices&h=shards,disk.indices,disk.used,disk.available,disk.total,disk.percent shards disk.indices disk.used disk.total disk.percent 123 137 179.4gb 185.9gb 242gb 76 Also from the command output, du -h / | sort -rh | head -10 181G /var/lib/elasticsearch/nodes/0/indices 181G /var/lib/elasticsearch/nodes/0 181G /var/lib/elasticsearch/nodes 181G /var/lib/elasticsearch Please suggest as this on production. Thanks, ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [January 6, 2024, 6:33pm UTC](https://discuss.elastic.co/t/does-single-node-elasticsearch-supports-ilm-ploicy/350124/15 "2024-01-06T18:33:04Z")
> [@Ravi\_Pattar](#): > > Please suggest as this on production. Hi @Ravi_Pattar Again this is your production system. I would highly recommend testing this on a non-production system. > [@Ravi\_Pattar](#): > > If this is a case then can I change it to lesser value other than 15 days e.g 5 or \< 10 days etc in the delete phase instead of waiting for another 15 days more because I can see that the disk space utilization has started piling up. Yes, of course, you can set delete to what you like... Perhaps a suggestion Rollover on 1 day... Then Adjust the Delete phase to what you want... in Days... this will give you a lot of control. After you set rollover to 1 fay, you will need to force a rollover on the current index so it does not wait to take effect. Shortening the Delete phase should work after it is set... Then you can optimize later (like Rollover on 3 Days) if needed. ---
**Author:** ![Ravi\_Pattar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ravi_pattar/32/124209_2.png) [@Ravi\_Pattar](https://discuss.elastic.co/u/Ravi_Pattar)\ **Post date:** [January 9, 2024, 7:43am UTC](https://discuss.elastic.co/t/does-single-node-elasticsearch-supports-ilm-ploicy/350124/16 "2024-01-09T07:43:23Z")
> [@stephenb](#): > > Yes, of course, you can set delete to what you like... > > Perhaps a suggestion > > Rollover on 1 day... > > Then Adjust the Delete phase to what you want... in Days... this will give you a lot of control. > > After you set rollover to 1 fay, you will need to force a rollover on the current index so it does not wait to take effect. > > Shortening the Delete phase should work after it is set... > > Then you can optimize later (like Rollover on 3 Days) if needed. Hi @stephenb I did few changes on my test environment. e.g. below On the Kibana GUI Stack Management \> Index Lifecycle policies and the filtered out for filebeat Applied the ILM for filebeat as below ```auto Hot phase (Required) Rollover: enabled Maximum primary shard size: 5G Maximum index size: 5G Maximum age: 1 days Delete phase Move data into phase when:2 days old ``` Current status when checked using the curl ```auto # curl -X GET "localhost:9200/_cat/indices/filebeat*?v" health status index uuid pri rep docs.count docs.deleted store.size pri.store.size yellow open filebeat-7.17.13-2023.12.29 bgI3JWqJQv-CF3URgOboXA 1 1 360230 0 40.7mb 40.7mb yellow open filebeat-7.17.13-2023.12.28 aN5saI_kRJmrXBTWhZ7aUg 1 1 157875 0 17.7mb 17.7mb yellow open filebeat-7.17.13-2023.12.27 3Ff2VmfjT72ElvbbUC3VyA 1 1 267396 0 29.8mb 29.8mb yellow open filebeat-7.17.13-2024.01.02 uuFIjZdkTYqO6RNEx9B9gg 1 1 4984 0 2.7mb 2.7mb yellow open filebeat-7.17.13-2024.01.03 MgKear7JQA6hYs5VdPdYxg 1 1 7668 0 2.4mb 2.4mb yellow open filebeat-7.17.13-2024.01.01 Kj2FCRBUQca5n3mg-yed7Q 1 1 2671 0 1.2mb 1.2mb yellow open filebeat-7.17.13-2024.01.04 85EFuavPQamUfnLCCu-Ozw 1 1 4507 0 2.6mb 2.6mb yellow open filebeat-7.17.13-2024.01.05 JbsEFg0tSJCGsJddNgbYgg 1 1 2928 0 1.5mb 1.5mb yellow open filebeat-7.17.13-2024.01.01-000005 9UnVNkJAS1aaUXEdrt2zZA 1 1 0 0 227b 227b yellow open filebeat-7.17.13-2023.12.26 MOlB9WyrSOaDDCf-CYJoJQ 1 1 2108355 0 235mb 235mb yellow open filebeat-7.17.13-2024.01.09-000006 oTXERMP3RFWyUDqy3DMj9A 1 1 0 0 227b 227b yellow open filebeat-7.17.13-2024.01.09 rIQ_RaOeSIuycyO-Twxizw 1 1 2621 0 1.5mb 1.5mb ``` Please suggest if there's anything wrong here and why I don't see delete is not occurring. Thanks, ---
**Author:** ![Ravi\_Pattar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ravi_pattar/32/124209_2.png) [@Ravi\_Pattar](https://discuss.elastic.co/u/Ravi_Pattar)\ **Post date:** [January 9, 2024, 1:11pm UTC](https://discuss.elastic.co/t/does-single-node-elasticsearch-supports-ilm-ploicy/350124/17 "2024-01-09T13:11:16Z")
Hello, Can someone please help on my recent post? Thanks, ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [January 9, 2024, 3:06pm UTC](https://discuss.elastic.co/t/does-single-node-elasticsearch-supports-ilm-ploicy/350124/18 "2024-01-09T15:06:00Z")
First Please be patient. There are many questions and your question is no more important than anyone else's. It's only been a couple hours. We will get to your questions if and when we can. There is no guarantee that your question will even get answered. It's only volunteers here. Run this for your policy Please run these show the command and the result `GET _ilm/policy/` Then run `GET filebeat-7.17.13-2023.12.29/_ilm/explain` ---
**Author:** ![Ravi\_Pattar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ravi_pattar/32/124209_2.png) [@Ravi\_Pattar](https://discuss.elastic.co/u/Ravi_Pattar)\ **Post date:** [January 9, 2024, 6:08pm UTC](https://discuss.elastic.co/t/does-single-node-elasticsearch-supports-ilm-ploicy/350124/19 "2024-01-09T18:08:15Z")
> [@stephenb](#): > > `GET _ilm/policy/` > > Hi Stephen, > > Thanks, please find the details below. > > GET \_ilm/policy/filebeat > > ```auto > { > "filebeat" : { > "version" : 13, > "modified_date" : "2024-01-09T12:23:33.729Z", > "policy" : { > "phases" : { > "hot" : { > "min_age" : "0ms", > "actions" : { > "rollover" : { > "max_size" : "50mb", > "max_primary_shard_size" : "49mb", > "max_age" : "1d" > } > } > }, > "delete" : { > "min_age" : "1d", > "actions" : { > "delete" : { > "delete_searchable_snapshot" : true > } > } > } > } > }, > "in_use_by" : { > "indices" : [ > "filebeat-7.17.13-2023.12.29", > "filebeat-7.17.13-2023.12.28", > "filebeat-7.17.13-2023.12.27", > "filebeat-7.17.13-2023.12.26" > ], > "data_streams" : [], > "composable_templates" : [] > } > } > } > > ``` > > Then run > > `GET filebeat-7.17.13-2023.12.29/_ilm/explain` ```auto { "indices" : { "filebeat-7.17.13-2023.12.29" : { "index" : "filebeat-7.17.13-2023.12.29", "managed" : true, "policy" : "filebeat", "lifecycle_date_millis" : 1703829688349, "age" : "11.49d", "phase" : "hot", "phase_time_millis" : 1704823122627, "action" : "rollover", "action_time_millis" : 1703829689068, "step" : "check-rollover-ready", "step_time_millis" : 1704823122627, "is_auto_retryable_error" : true, "failed_step_retry_count" : 94, "phase_execution" : { "policy" : "filebeat", "phase_definition" : { "min_age" : "0ms", "actions" : { "rollover" : { "max_size" : "50mb", "max_primary_shard_size" : "49mb", "max_age" : "1d" } } }, "version" : 13, "modified_date_in_millis" : 1704803013729 } } } } ``` Thanks, ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [January 9, 2024, 6:34pm UTC](https://discuss.elastic.co/t/does-single-node-elasticsearch-supports-ilm-ploicy/350124/20 "2024-01-09T18:34:58Z")
> [@Ravi\_Pattar](#): > > ```auto > "step" : "check-rollover-ready", > "step_time_millis" : 1704823122627, > "is_auto_retryable_error" : true, > "failed_step_retry_count" : 94, > > ``` This index is Failing rollover... perhaps you changed the index name ... to daily indices... Are you ingesting directly from Filebeat to Elasticsearch? Share your entire filebeat.yml please... [Next page](https://discuss.elastic.co/t/does-single-node-elasticsearch-supports-ilm-ploicy/350124.md?page=2)