Defined ILM policy not working

Elastic Stack Elasticsearch
1

Hi All,

I hope are doing well!

We are running Elastic 7.15 version with a single node.
Currently the data in our winlogbeat index is all dated from back February 2021 hence we defined an ILM policy as below though please note I changed the max age from 30 days to 90 days as earlier as well ILM was not rolling over thinking removing the policy and reapplying will help

{
  "winlogbeat" : {
    "version" : 5,
    "modified_date" : "2021-11-02T09:08:40.424Z",
    "policy" : {
      "phases" : {
        "hot" : {
          "min_age" : "0ms",
          "actions" : {
            "rollover" : {
              "max_size" : "800gb",
              "max_primary_shard_size" : "800gb",
              "max_age" : "90d"
            },
            "set_priority" : {
              "priority" : 100
            }
          }
        },
        "delete" : {
          "min_age" : "95d",
          "actions" : {
            "delete" : {
              "delete_searchable_snapshot" : true
            }
          }
        }
      }
    },
    "in_use_by" : {
      "indices" : [
        "winlogbeat-7.8.0-000001",
        "winlogbeat-7.8.0-000002"
      ],
      "data_streams" : [ ],
      "composable_templates" : [ ]
    }
  }
}

And below is the current status of indices

{
  "indices" : {
    "winlogbeat-7.8.0-000001" : {
      "index" : "winlogbeat-7.8.0-000001",
      "managed" : true,
      "policy" : "winlogbeat",
      "lifecycle_date_millis" : 1635155559372,
      "age" : "8.95d",
      "phase" : "hot",
      "phase_time_millis" : 1635843483018,
      "action" : "complete",
      "action_time_millis" : 1635843647016,
      "step" : "complete",
      "step_time_millis" : 1635843647016,
      "phase_execution" : {
        "policy" : "winlogbeat",
        "phase_definition" : {
          "min_age" : "0ms",
          "actions" : {
            "rollover" : {
              "max_size" : "800gb",
              "max_primary_shard_size" : "800gb",
              "max_age" : "90d"
            },
            "set_priority" : {
              "priority" : 100
            }
          }
        },
        "version" : 5,
        "modified_date_in_millis" : 1635844120424
      }
    },
    "winlogbeat-7.8.0-000002" : {
      "index" : "winlogbeat-7.8.0-000002",
      "managed" : true,
      "policy" : "winlogbeat",
      "lifecycle_date_millis" : 1635155558370,
      "age" : "8.95d",
      "phase" : "hot",
      "phase_time_millis" : 1635764162691,
      "action" : "rollover",
      "action_time_millis" : 1635155561373,
      "step" : "check-rollover-ready",
      "step_time_millis" : 1635764162691,
      "is_auto_retryable_error" : true,
      "failed_step_retry_count" : 246,
      "phase_execution" : {
        "policy" : "winlogbeat",
        "phase_definition" : {
          "min_age" : "0ms",
          "actions" : {
            "rollover" : {
              "max_size" : "800gb",
              "max_primary_shard_size" : "800gb",
              "max_age" : "90d"
            },
            "set_priority" : {
              "priority" : 100
            }
          }
        },
        "version" : 5,
        "modified_date_in_millis" : 1635844120424
      }
    }
  }
}

I was expecting the ILM to create a new indice every 30 or 90 days as what we define in ILM max age policy as we want to delete anything older than 90 days and hence have set it for autodeletion at 95 days but it does not seem to work.

2

any suggestion or guidance on how to fix this?
Thanks.

3

Hi,

Any suggestions on fixing this.
Thanks.

4

Thanks for using Elasticsearch and ILM.

ILM will trigger a rollover for the managed indices when any of the conditions defined in the rollover action is met (first one to be met triggers to rollover).

"rollover" : {
          "max_size" : "800gb",
          "max_primary_shard_size" : "800gb",
          "max_age" : "90d"
        }

This definition will rollover the managed index if the sum of the primary shards for the index exceeds 800gb, or if any primary shard size exceeds 800gb. or if the index is older than 90d.

Your index was probably rolled over due to the max_primary_shard_size condition being triggered. You can check this using the index stats API

With your current policy definition the index will be deleted 95 days after it was rolled over so your data retention requirements are still met.

5

Thank you @andreidan & really appreciate for checking on this!
Initially the rollover policy had defined "max_size" : "200gb", with "max_primary_shard_size" : "200gb", and "max_age" : "30d".
Hence new indices should have been created every 30 days or with current scenarioevery 90 days or reaching 800gb whichever earlier but we could see all the documents/events saved in the index winlogbeat-7.8.0-000001.

Are we doing something wrong here?

Thank you once again for helping on this.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.