There is a way to set the time of the day when the index rollover will happen?

Elastic Stack Elasticsearch
1

I have an index template like the following:

{
  "index": {
    "lifecycle": {
      "name": "indexname_ilm_policy",
      "rollover_alias": "indexname"
    },
    "number_of_replicas": "2",
    "refresh_interval": "60s"
  }
}

And the current ILM policy:

{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {
          "rollover": {
            "max_age": "1d"
          },
          "set_priority": {
            "priority": 100
          }
        }
      },
      "warm": {
        "min_age": "1d",
        "actions": {
          "set_priority": {
            "priority": 50
          }
        }
      },
      "delete": {
        "min_age": "120d",
        "actions": {
          "delete": {
            "delete_searchable_snapshot": true
          }
        }
      }
    }
  }
}

So every day a new indexname-0000* is created around 7:00 AM, which is a time when the customers are already indexing a lot of data, and sometimes there are spikes in the service response time at this exact time.

My question is: There is a way to set the time of the day when the index rollover will happen and then indexes would be created at a time with less indexing activity like in the dawn around 4:00 AM.

I didn't find any docs related to it. My elasticsearch cluster version is 7.15.2.
Thanks in advance.

2

You can set a specific time, but if your ILM policy only rollover based on the age, you may force a manual rollover at the time you want and the next rollover will happen 1 day after that date.

So, I think that if you manually rollover your data stream at 4:00 AM, the next data stream will rollover at this date since it will calculate based on the creation date.

1 Like
3

Thanks @leandrojmp! It makes sense. I'll try to perform that and if it works I will set your reply as solution!

4

@leandrojmp I did a test in our QA environment but the strategy doesn't work. The rollover created an index with the new time but on the next day, the new index was created at the old time.

I think it is because the warm phase minimum age of 1 day.

How can we set the specific rollover time?

5

Can you share some evidence of this? If you manually rollover an index, then every other action in the ILM for new indices will be based on this.

For example, if I rollover an index now, 14:00 UTC, and I have a rollover of 1 day, this index should rollover tomorrow around 14:00 UTC, and the warm phase will also be based on this time.

But even in this way, since ILM can take up to 20 minutes to execute an action, this time would shift over time.

Just fixing a typo on my previous answer that I just saw now, you can't set a specific time for ILM to execute a rollover.

1 Like
6

I put down here in the below JSON the last 4 created indexes information, pay attention to the creation date field, I converted the millis for a more legible date and time in the comments.

{
  "indexname-000392": {
    "aliases" : {
      "indexname" : {
        "is_write_index" : false
      }
    },
    "settings" : {
      "index" : {
        "lifecycle" : {
          "name" : "indexname_ilm_policy",
          "rollover_alias" : "indexname",
          "indexing_complete" : "true"
        },
        "routing" : {
          "allocation" : {
            "include" : {
              "_tier_preference" : "data_warm,data_hot"
            }
          }
        },
        "refresh_interval" : "30s",
        "number_of_shards" : "1",
        "provided_name" : "indexname-000392",
        "creation_date" : "1702017703384", //Fri 8 December 2023 03:41:43
        "priority" : "50",
        "number_of_replicas" : "2",
        "uuid" : "9Ehm_Pb3SHO3m6sjvDYtzA",
        "version" : {
          "created" : "7150299"
        }
      }
    }
  },
  "indexname-000393" : { // Here I did the manual rollover at Fri 8 December 2023 12:30:15
    "aliases" : {
      "indexname" : {
        "is_write_index" : false
      }
    },
    "settings" : {
      "index" : {
        "lifecycle" : {
          "name" : "indexname_ilm_policy",
          "rollover_alias" : "indexname",
          "indexing_complete" : "true"
        },
        "routing" : {
          "allocation" : {
            "include" : {
              "_tier_preference" : "data_warm,data_hot"
            }
          }
        },
        "refresh_interval" : "30s",
        "number_of_shards" : "1",
        "provided_name" : "indexname-000393",
        "creation_date" : "1702049415650", // Fri 8 December 2023 12:30:15
        "priority" : "50",
        "number_of_replicas" : "2",
        "uuid" : "oKY9uR0LSNKdny_8QB-_Ng",
        "version" : {
          "created" : "7150299"
        }
      }
    }
  }, // Obs: No index created at 9 and 10 of December due to QA environment shutdown on weekend
  "indexname-000394" : {
    "aliases" : {
      "indexname" : {
        "is_write_index" : false
      }
    },
    "settings" : {
      "index" : {
        "lifecycle" : {
          "name" : "indexname_ilm_policy",
          "rollover_alias" : "indexname",
          "indexing_complete" : "true"
        },
        "routing" : {
          "allocation" : {
            "include" : {
              "_tier_preference" : "data_content"
            }
          }
        },
        "refresh_interval" : "30s",
        "number_of_shards" : "1",
        "provided_name" : "indexname-000394",
        "creation_date" : "1702275102493", // Mon 11 December 2023 03:11:42
        "priority" : "100",
        "number_of_replicas" : "2",
        "uuid" : "AhAyHrwORtOgyPuLiSFdfg",
        "version" : {
          "created" : "7150299"
        }
      }
    }
  },
  "indexname-000395" : {
    "aliases" : {
      "indexname" : {
        "is_write_index" : true
      }
    },
    "settings" : {
      "index" : {
        "lifecycle" : {
          "name" : "indexname_ilm_policy",
          "rollover_alias" : "indexname"
        },
        "routing" : {
          "allocation" : {
            "include" : {
              "_tier_preference" : "data_content"
            }
          }
        },
        "refresh_interval" : "30s",
        "number_of_shards" : "1",
        "provided_name" : "indexname-000395",
        "creation_date" : "1702362094936", // Tue 12 December 2023 03:21:34
        "priority" : "100",
        "number_of_replicas" : "2",
        "uuid" : "O7nCyOtjSkSI4EHU1dv9MA",
        "version" : {
          "created" : "7150299"
        }
      }
    }
  }  
}

I suspect that the manually created index: indexname-000393 doesn't have the minimum age of 1 day from the warm phase to rollover and then the ILM policy rollover the indexname-000392 which already has this age.

What you think?

7

No, the rollover is incremental, if you manually executed the rollover for the indexname-000392, then the indexname-000393 was createdf and after 1 day it would rolled over into indexname-000394.

But it seems that you shutdown your environment before the indexname-000393 completed 1 day, when this happens it will only rollover when your environment starts again.

I'm assuming that the QA environment was turned on again on monday, close to this time:

"creation_date" : "1702275102493", // Mon 11 December 2023 03:11:42

Is that right? If so, everything is working as expected, as soon as Elasticsearch started it checked the ILM of every index to see the actions that need to be performed, then it saw that the index indexname-000393 was more than 1 day older and needed a rollover.

If you shutdown your environment every friday, you need to perform a manual rollover now and check tomorrow to see when the index will be created.

8

The shutdown could be the reason, but I have to investigate. I will ask the guys the time that QA environment starts because the time the indexname-000393 did the rollover is very close to the time of the olds index before the manual rollover. Also I will perform another manual rollover on indexname-000395 to check tomorrow.

Thanks in advance again @leandrojmp

9

It's confirmed @leandrojmp I rolled over the index indexname-000395 at nearly 11:57 into indexname-000396 and today the indexname-000396 was created at 12.

You were right. Thanks again! :clap:t4:

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.