Index Rollover not working after reaching maxday

Elastic Stack Elasticsearch
1

Not able to rollover after 1 day
I have set parse_origination_date to true
the index is like access-log-appname-2022.07.06-000001

{
  "policy":{
    "phases":{
      "hot":{
        "min_age":"0ms",
        "actions":{
          "rollover":{
            "max_age":"1d"
          },
          "set_priority":{
            "priority":100
          }
        }
      },
      "warm":{
        "min_age":"1d",
        "actions":{
          "set_priority":{
            "priority":50
          }
        }
      },
      "delete":{
        "min_age":"2d",
        "actions":{
          "delete":{
            "delete_searchable_snapshot":true
          }
        }
      }
    }
  }
}

{
  "indices": {
    "access-log-w302-2022.07.06-000001": {
      "index": "access-log-appname-2022.07.06-000001",
      "managed": true,
      "policy": "7-days-access-log",
      "lifecycle_date_millis": 1657065600000,
      "age": "1.08d",
      "phase": "hot",
      "phase_time_millis": 1657095639572,
      "action": "rollover",
      "action_time_millis": 1657095639972,
      "step": "check-rollover-ready",
      "step_time_millis": 1657095639972,
      "phase_execution": {
        "policy": "7-days-access-log",
        "phase_definition": {
          "min_age": "0ms",
          "actions": {
            "set_priority": {
              "priority": 100
            },
            "rollover": {
              "max_age": "1d"
            }
          }
        },
        "version": 1,
        "modified_date_in_millis": 1657095074154
      }
    }
  }
}

{
  "template": {
    "settings": {
      "index": {
        "lifecycle": {
          "name": "7-days-access-log",
          "parse_origination_date": "true",
          "rollover_alias": "access-log"
        },
        "number_of_shards": "1"
      }
    },
    "mappings": {
      "properties": {
        "@timestamp": {
          "type": "date"
        },
        "aa": {
          "type": "text"
        }
      }
    },
    "aliases": {}
  }
}

I expect it will create a new index access-log-appname-2022.07.07-000001 after it reach the max day.
or I understand it wrong, plz help

2

Is the name of the policy you showed 7-days-access-log (I do not see anything related to 7 days in it...)?

The sequence number is monotonically incrementing and not per day, so the next index would end in -000002.

3

oh, I changed the delete min_age from 7d to 2d

and I think it is not using origination_date
it created a new index access-log-appName-2022.07.07-000002 at 2022-07-07 16:20:41 instead of 2022-07-07 00:00:00

{
  "indices": {
    "access-log-w302-2022.07.06-000001": {
      "index": "access-log-appName-2022.07.06-000001",
      "managed": true,
      "policy": "7-days-access-log",
      "lifecycle_date_millis": 1657065600000,
      "age": "1.41d",
      "phase": "warm",
      "phase_time_millis": 1657182041364,
      "action": "complete",
      "action_time_millis": 1657182041764,
      "step": "complete",
      "step_time_millis": 1657182041764,
      "phase_execution": {
        "policy": "7-days-access-log",
        "phase_definition": {
          "min_age": "1d",
          "actions": {
            "set_priority": {
              "priority": 50
            }
          }
        },
        "version": 1,
        "modified_date_in_millis": 1657095074154
      }
    }
  }
}
4

Rollover is based on the index creation date and not the date you put in the name. If the 000001 index was created around 16:20:41 the rollover to the 000002 should happen approximately 24 hours later if you have configured rollover for 1 day. What you are seeing is therefore expected.

The whole point of rollover is that you will roll over to a new index based on age or size, as this will allow you to achieve a reasonably uniform shard size. This was never designed to roll over at specific times as that would defeat the main purpose. If you want strictly day based indices, determine the indexname based on the timestamp before you index and do not use rollover.

5

but I have set index.lifecycle.parse_origination_date to true

will the rollover action use origination_date of the index as begin time?

6

Yes. If you want indices to cover specific time periods, do not use rollover.

Before rollover was available using indices covering fixed time periods based on the index name was the standard. This is still available and still supported by Logstash and may be better for you.

7

I was trying to use rollover to delete specific time period indices after N days.
thought they can use together.

8

Time-based indices rely on managing data retention by deleting complete indices. If you have a specific target retention period you are therefore likely to always have a certain amount of data in the cluster that has exceeded the retention period. In many use cases users require at least X days of data to be available, but do not mind if a little extra data is available from time to time. This is where rollover shines - as long as you do not need to control the deletion and exact amount of extra data available in detail you can let indices cover more flexible time periods and better control their size, which is quite important for optimum performance.

If you are not willing to relinquish this control, rollover and the benefits it brings might not be for you.

1 Like
9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.