ILM - Does not trigger rollover

[grumo35]

Hi,

I've configured ILM on datastream and a rollover policy.

The goal is to have my datastream backing indices to rollover daily.

The ILM is configured as follow :

Two weeks passed, still on the same backing indices i decided to trigger-it manually.

POST my-data-stream/_rollover

It works but not really as intended my indices went from .ds-my-data-stream-2023.XX.XX-00000001 to .ds-my-data-stream-2023.XX.XX-00000002.

I've seen from the documentation that you can do date math, and i'm pretty sure the indices is recreated after the delete phase on the date of the day.

What am i missing to trigger rollover daily ? and set the new indices name to the current date ?

[andreidan]

Thanks for using Elasticsearch and ILM.

Did your index contain any documents? ILM will not rollover empty indices even if the other criteria (e.g. max_age) is met.

More details here and workaround : Rollover | Elasticsearch Guide [8.11] | Elastic

Empty indices will not be rolled over, even if they have an associated max_age that would otherwise result in a roll over occurring. A policy can override this behavior, and explicitly opt in to rolling over empty indices, by adding a "min_docs": 0 condition. This can also be disabled on a cluster-wide basis by setting indices.lifecycle.rollover.only_if_has_documents to false.

[grumo35]

Hello thanks for your kind reply,

The index was not empty.

Here you can see when i created the data-stream using the date of the first backing index.

Then i waited for the ILM policy delete phase to be triggered which worked, but i'd love to have daily backing indices.

Yesterday i've triggered the 2023.12.06 using the POST API rollover on the datastream.

It looks like it created the 2023.12.07

Should i trigger the rollover on every datastream using the POST API ? Where could i check this setting on other datastream ?

[stephenb (Stephen Brown)]

What version are you on?

Perhaps run

GET <index>/_ilm/explain

As that should provide insight of what's going on.

Daily backing indices are definitely supported / work I use them all the time.

I suspect there was some initial issue.. like you initially created the ILM policy with defaults and then changed/updated the policy and perhaps that update did not apply to the initial index. There is a way to check with the _ilm/explain and then look at the details of the ILM policy.

It should work going forward.

If you have any of the "stuck" indices run explain and lets look

[grumo35]

Hey,

Yes absolutely so the syslog datastream where i've triggered manually seems to be doing "fine"

Here is the ilm explain of the current backing index :

{
  "indices" : {
    ".ds-filebeat-linux-syslog-2023.12.07-000003" : {
      "index" : ".ds-filebeat-linux-syslog-2023.12.07-000003",
      "managed" : true,
      "policy" : "30d",
      "lifecycle_date_millis" : 1701944834271,
      "age" : "5.63h",
      "phase" : "hot",
      "phase_time_millis" : 1701944834434,
      "action" : "rollover",
      "action_time_millis" : 1701944834434,
      "step" : "check-rollover-ready",
      "step_time_millis" : 1701944834434,
      "phase_execution" : {
        "policy" : "30d",
        "phase_definition" : {
          "min_age" : "0ms",
          "actions" : {
            "rollover" : {
              "max_primary_shard_size" : "50gb",
              "max_age" : "1d"
            }
          }
        },
        "version" : 3,
        "modified_date_in_millis" : 1701181053442
      }
    }
  }
}

We can see that it seems to work pretty fine

However another data stream where i didnt ran the POST rollover query

{
  "indices" : {
    ".ds-filebeat-xx-2023.11.25-000001" : {
      "index" : ".ds-filebeat-xx-2023.11.25-000001",
      "managed" : true,
      "policy" : "30d",
      "lifecycle_date_millis" : 1700927237089,
      "age" : "12.03d",
      "phase" : "hot",
      "phase_time_millis" : 1700927237227,
      "action" : "complete",
      "action_time_millis" : 1700927237026,
      "step" : "complete",
      "step_time_millis" : 1700927237227,
      "phase_execution" : {
        "policy" : "30d",
        "phase_definition" : {
          "min_age" : "0ms",
          "actions" : { }
        },
        "version" : 1,
        "modified_date_in_millis" : 1698406220853
      }
    }
  }
}

We can see that the actions rollover is not present depsite having the same ILM policy ?

[grumo35]

I'm in 7.17, And yes i've changed this ILM policy many times and enabled / disabled rollover !

I'm still in testing phase, some concepts are hard to get a nice grasp on

What can help me to make sure everything is triggered ?

Also silly question but could i configure the rollover to happen at midnight ? not very important but cleaner for my organization.

Thanks for your help !

[stephenb (Stephen Brown)]

Hi @grumo35
Its all right there....
This index is still on version 1 of the ILM policy you can see the rollover is 30d..
So, you changed the ILM policy to 1 Day After this index was created...
That is why the new policy has not been picked up.
I think the behavior is better in 8.X and the policy will get picked up.
There is a lot of subtly of when the new version is picked up... and I think that behavior is better in 8.x

So that is your probably today.. New Indices should pick up the new policy

It is interesting you can change the further down retention policies like days in warm / delete etc... those should get picked up if I recall... but since you are already in HOT phase it operates on the policy it "saw" when it was created.

@grumo35

      "phase_execution" : {
        "policy" : "30d",<!---- HERE
        "phase_definition" : {
          "min_age" : "0ms",
          "actions" : { }
        },
        "version" : 1, <!---- HERE
        "modified_date_in_millis" : 1698406220853
      }
    }
  }
}

[grumo35]

Thanks for letting me know the version thing !!!

let's rollover now