Control phase transition timings in ILM

Elastic Stack Elasticsearch
1

I'm trying to implement an ilm policy where the index will rollover every month or if the size reached 50GB.
And since by default, ilm based the age in index creation date, and creating the index mid december means, we need to adjust and enable the parse_origination_date.

Here's our ILM Policy

{
  "rollover-monthly-keep-90d" : {
    "version" : 2,
    "modified_date" : "2021-12-16T12:41:28.721Z",
    "policy" : {
      "phases" : {
        "hot" : {
          "min_age" : "0ms",
          "actions" : {
            "rollover" : {
              "max_size" : "50gb",
              "max_age" : "30d"
            },
            "set_priority" : {
              "priority" : 100
            }
          }
        },
        "delete" : {
          "min_age" : "90d",
          "actions" : { }
        }
      }
    }
  }
}

Our Index Template

{
  "vsphere-stats-v3" : {
    "order" : 0,
    "index_patterns" : [
      "vsphere-stats-v3-*"
    ],
    "settings" : {
      "index" : {
        "lifecycle" : {
          "name" : "rollover-monthly-keep-90d",
          "parse_origination_date" : "true",
          "rollover_alias" : "vsphere-stats-v3"
        },
        "number_of_shards" : "4",
        "number_of_replicas" : "1"
      }
    },
    "mappings" : { },
    "aliases" : { }
  }
}

We used the ff output in logstash

	elasticsearch {
        ilm_rollover_alias => "vsphere-stats-v3"
        ilm_pattern => "vsphere-stats-v3-{now/d}-000001"
        ilm_policy => "rollover-monthly-keep-90d"
        hosts => ["Host1", "Host2", "Host3"]
      }

But upon checking the indices created after a month, it did rollover but the index age is same for both indices. This is critical because it will delete all the indices at once after 90days.

"indices" : {
    "vsphere-stats-v3-2021.12.01-000001" : {
      "index" : "vsphere-stats-v3-2021.12.01-000001",
      "managed" : true,
      "policy" : "rollover-monthly-keep-90d",
      "lifecycle_date_millis" : 1638316800000,
      "age" : "50.3d",
      "phase" : "hot",
      "phase_time_millis" : 1639720820792,
      "action" : "complete",
      "action_time_millis" : 1642313154868,
      "step" : "complete",
      "step_time_millis" : 1642313154868,
      "phase_execution" : {
        "policy" : "rollover-monthly-keep-90d",
        "phase_definition" : {
          "min_age" : "0ms",
          "actions" : {
            "rollover" : {
              "max_size" : "50gb",
              "max_age" : "30d"
            },
            "set_priority" : {
              "priority" : 100
            }
          }
        },
        "version" : 2,
        "modified_date_in_millis" : 1639658488721
      }
    },
    "vsphere-stats-v3-2021.12.01-000002" : {
      "index" : "vsphere-stats-v3-2021.12.01-000002",
      "managed" : true,
      "policy" : "rollover-monthly-keep-90d",
      "lifecycle_date_millis" : 1638316800000,
      "age" : "50.3d",
      "phase" : "hot",
      "phase_time_millis" : 1642313153996,
      "action" : "rollover",
      "action_time_millis" : 1642313154955,
      "step" : "check-rollover-ready",
      "step_time_millis" : 1642313154955,
      "phase_execution" : {
        "policy" : "rollover-monthly-keep-90d",
        "phase_definition" : {
          "min_age" : "0ms",
          "actions" : {
            "rollover" : {
              "max_size" : "50gb",
              "max_age" : "30d"
            },
            "set_priority" : {
              "priority" : 100
            }
          }
        },
        "version" : 2,
        "modified_date_in_millis" : 1639658488721
      }
    }
  }
}

Checking the minimum and max date of the latest index

image

What do you think?

2

anyone from the support?

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.