Index Doesnt move to DELETE Phase

Elastic Stack Elasticsearch
1

Hello everyone

I have problems with ILMs

My index doesn't move to from HOT Phase to DELETE Phase

this is my ilm policy which I expect to being rolled over either 30 days old or 15gb limit

{
  "30d-rollover-50d-delete": {
    "version": 4,
    "modified_date": "2020-07-16T09:24:17.682Z",
    "policy": {
      "phases": {
        "hot": {
          "min_age": "0ms",
          "actions": {
            "rollover": {
              "max_age": "30d",
              "max_size": "15gb"
            },
            "set_priority": {
              "priority": 100
            }
          }
        },
        "delete": {
          "min_age": "50d",
          "actions": {
            "delete": {
              "delete_searchable_snapshot": true
            }
          }
        }
      }
    },
    "in_use_by": {
      "indices": [
        "my-index-000004",
        "my-index-000005",
      ]
    }
  }
}

This is my index settings for my-index-000004

{
  "indices": {
    "my-index-000004": {
      "index": "my-index-000004",
      "managed": true,
      "policy": "30d-rollover-50d-delete",
      "index_creation_date_millis": 1726403363137, -> 2024-02-01 04:42:43 UTC
      "time_since_index_creation": "53.95d",
      "lifecycle_date_millis": 1728995963326,
      "age": "23.95d",
      "phase": "hot",
      "phase_time_millis": 1726403363234,
      "action": "complete",
      "action_time_millis": 1728995963926,
      "step": "complete",
      "step_time_millis": 1728995963926,
      "phase_execution": {
        "policy": "30d-rollover-50d-delete",
        "phase_definition": {
          "min_age": "0ms",
          "actions": {
            "rollover": {
              "max_age": "30d",
              "min_docs": 1,
              "max_primary_shard_docs": 200000000,
              "max_size": "15gb"
            },
            "set_priority": {
              "priority": 100
            }
          }
        },
        "version": 4,
        "modified_date_in_millis": 1594891457682
      }
    }
  }
}

This is my index settings for my-index-000005

{
  "indices": {
    "my-index-000005": {
      "index": "my-index-000005",
      "managed": true,
      "policy": "30d-rollover-50d-delete",
      "index_creation_date_millis": 1728995963321, -> 2024-03-18 04:32:43 UTC
      "time_since_index_creation": "23.95d",
      "lifecycle_date_millis": 1728995963321,
      "age": "23.95d",
      "phase": "hot",
      "phase_time_millis": 1728995963726,
      "action": "rollover",
      "action_time_millis": 1728995964126,
      "step": "check-rollover-ready",
      "step_time_millis": 1728995964126,
      "phase_execution": {
        "policy": "30d-rollover-50d-delete",
        "phase_definition": {
          "min_age": "0ms",
          "actions": {
            "rollover": {
              "max_age": "30d",
              "min_docs": 1,
              "max_primary_shard_docs": 200000000,
              "max_size": "15gb"
            },
            "set_priority": {
              "priority": 100
            }
          }
        },
        "version": 4,
        "modified_date_in_millis": 1594891457682
      }
    }
  }
}

Sizes of my indices

index              store.size
my-index-000005    103.5mb
my-index-000004    1.6gb

As you can see between index creation days there are 46 days

  • Date 1: February 1, 2024 - > my-index-000004
  • Date 2: March 18, 2024 -> my-index-000005

Why it doesn't roll over when it reaches 30 days ?

I also realized such problem in other ILMs for sizes as well but let's focus on one case only for now

3

Hi @willymathew Welcome to the community.

Let's see if we can work through this

I am using this tool, https://www.epochconverter.com/, to check the dates. It does not agree with what you have provided so I am not sure how your are doing the conversions,

so let's look at the dates and explain what is going on.

Below is not your index settings it is the result of running

GET my-index-000004/_ilm/explain

And so you should look the _ilm/explain docs to understand the data

This is the most crucial concept, often overlooked, and it very frequently confuses.

If an index has been rolled over, then the min_age value is relative to the time the index was rolled over, not the index creation time. Learn more.

So, looking at your first example...

{
  "indices": {
    "my-index-000004": {
      "index": "my-index-000004",
      "managed": true,
      "policy": "30d-rollover-50d-delete",
      "index_creation_date_millis": 1726403363137,  << September 15, 2024 12:29:23.137 PM UTC
      "time_since_index_creation": "53.95d",  <<< Makes send 30 Days to Rollover plus 23.95 in hot
                                                  waiting to be 50 Days additional to be deleted. 
      "lifecycle_date_millis": 1728995963326,  << October 15, 2024 12:39:23.326 PM
      "age": "23.95d",  << Age in hot phase after rollover, needs to get to 50 to be deleted 
      "phase": "hot",
      "phase_time_millis": 1726403363234,  << September 15, 2024 12:29:23.234 PM
      "action": "complete",
      "action_time_millis": 1728995963926, << October 15, 2024 12:39:23.926 PM
      "step": "complete",
      "step_time_millis": 1728995963926, << October 15, 2024 12:39:23.926 PM
      "phase_execution": {
        "policy": "30d-rollover-50d-delete",
        "phase_definition": {
          "min_age": "0ms",
          "actions": {
            "rollover": {
              "max_age": "30d",
              "min_docs": 1,
              "max_primary_shard_docs": 200000000,
              "max_size": "15gb"
            },
            "set_priority": {
              "priority": 100
            }
          }
        },
        "version": 4,
        "modified_date_in_millis": 1594891457682
      }
    }
  }
}
  • After 30 Days or 15GB (nowhere close) the index will Rollover
  • Index Rolled Over after 30 Days on 0ct 15th
  • Then your policy says to wait 50 days after rollover, then delete it.
  • It is only 23.95 Days into that 50 Days.
  • So if your Index took 30 Days to Rollover then 50 Days for Delete then it will get deleted about 80 Days from creation.

This is what that says... and so it looks like it is operating correctly to me.

  • If your index only took 1 day to roll over (because fast / big ingestion), then it would spend 50 Days in Hot waiting to be deleted so about 51 Days from creation

So the question is, what do you want to have for your lifecycle?

There are some advanced setting which you could look at but if you are just getting started I would probably not recommend them.