ILM not executing DELETE phase

[radw (Radu Stefan)]

Hi all!
I'm using ELK stack v7.6.2 and v7.4.0. The same behaviour is happening on both versions.
I've created an ILM policy together with timeseries indices and a template for an index.
My ILM is rolling over my index at 40GB or 30 days, the template creates a new index afterwards and sets it as the write index.
I use an index alias to write to these time series indices and all goes well.
I'm trying to achieve the deletion of these indices 91 days after the rollover but this phase of the ILM never gets to be triggered.
At first I've created the ILM without the delete phase and I've added it afterwards but I guess this should not be a problem as I've played around with the rollover phase multiple times and every time I've got the desired result(eg: changing max_size or max_age for rollover).
My policy settings are as follows:

   {
   "policy_index_infra" : {
     "version" : 3,
     "modified_date" : "2020-10-02T04:53:07.209Z",
     "policy" : {
       "phases" : {
         "hot" : {
           "min_age" : "0ms",
           "actions" : {
             "rollover" : {
               "max_size" : "40gb",
               "max_age" : "30d"
             }
           }
         },
         "delete" : {
           "min_age" : "91d",
           "actions" : {
             "delete" : { }
           }
         }
       }
     }
   }
 }

The index ILM which should have been deleted after 91days from rollover is the following:

 {
   "indices" : {
     "index_infra-2020-07-30-000068" : {
       "index" : "index_infra-2020-07-30-000068",
       "managed" : true,
       "policy" : "policy_index_infra",
       "lifecycle_date_millis" : 1596303675069,
       "age" : "99.88d",
       "phase" : "completed",
       "phase_time_millis" : 1596303676222,
       "action" : "completed",
       "action_time_millis" : 1596303676222,
       "step" : "completed",
       "step_time_millis" : 1596303676222,
       "phase_execution" : {
         "policy" : "policy_index_infra",
         "version" : 2,
         "modified_date_in_millis" : 1583359226294
       }
     }
   }
 }

What I've tried:
I've created a new ILM policy with only the delete phase and it triggered correctly after a few minutes but it was based on creation time not on rollover time since I didn't have the rollover set up in that ILM policy.
I've waited even 112 days and still nothing happened so I've manually deleted the index.
I'm at dead end here since I've read every article on this forum and I've come to no conclusion.
Any help about where to look or what to test would be appreciated.

Cheers,
Radu

[dakrone (Lee Hinman)]

Hi Radu,

I notice that you mentioned:

At first I've created the ILM without the delete phase and I've added it afterwards but I guess this should not be a problem ..

This is actually what happened, before the policy had the delete phase added, this index moved to the terminal step (completed/completed/completed). To fix this you can do:

POST /_ilm/move/index_infra-2020-07-30-000068
{
  "current_step": { 
    "phase": "completed",
    "action": "completed",
    "name": "completed"
  },
  "next_step": { 
    "phase": "delete",
    "action": "delete",
    "name": "delete"
  }
}

This behavior is understandably less than ideal, which is why this behavior has been rectified in https://github.com/elastic/elasticsearch/pull/51631 (7.7+), where policies will stop in the last configured phase, so a delete phase added after will always be executed.

Hope this helps.