Inconsistency in data stream rollover

We want hourly rollover of the index and then index should get deleted after 12 hour.

To achieve this we have created data stream with a ILM policy which rollover index every 1hr and then the index goes into the warm phase and after 12 hour index goes into deleted phase.

Setup:
- Elasticsearch 8.6 docker on ubuntu 22.4
- We have setup a test cron job which push data to the index in every 5th min.
- Below are the scripts for the ILM policy and index template

	PUT _ilm/policy/ilm_test_activities2
	{
		"policy": {
			"phases": {
				"hot": {
					"min_age": "0s",
					"actions": {
						"set_priority": {
							"priority": 100
						},
						"rollover": {             
							"max_age": "1h"            
						},
					}
				},
				"warm": {
					"min_age": "0s",
					"actions": {
						"set_priority": {
							"priority": 50
						}
					}
				},
				"delete": {
					"min_age": "12h",
					"actions": {
					"delete": {}
					}
				}
			}
		}
	}
		
	PUT /_index_template/it_test_activities2
	{
		"index_patterns" : ["test-activity2-*"],
		"priority" : 1,
		"data_stream": { },
		"template": {
			"settings" : {
				"index" : {
					"lifecycle" : {
						"name" : "ilm_test_activities2"
					},
					"default_pipeline": "ip_timestamp",
					"number_of_replicas": 0
				}
			}
		}
	}

Expected behaviour for index rollover:
-Every hour new index should get created.

Current behaviour:
Index is not getting rollover every hour.
Test result:

  • First index created at: 07:45
  • Second index rollover : 08:51 // after 1 hr 06 mins
  • Third index rolleover : 10:01 // after 1 hr 10 mins
  • Fourth index rolleover: 11:11 // after 1 hr 10 mins
  • Fifth index rolleover : 13:01 // after 1 hr 50 mins
  • sixth index rolleover : 14:31 // after 1 hr 30 mins

We have observed more time gap(upto 3-5 hr) in the index rollover.

Is there any misconfiguration from our end?

Data stream information:

{
    ".ds-test_ilm_index-2023.01.20-000001": {
      "aliases": {},
      "mappings": {
        "_data_stream_timestamp": {
          "enabled": true
        },
        "properties": {
          "@timestamp": {
            "type": "date"
          },
          "test": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          }
        }
      },
      "settings": {
        "index": {
          "lifecycle": {
            "name": "ilm_test_activities",
            "indexing_complete": "true"
          },
          "routing": {
            "allocation": {
              "include": {
                "_tier_preference": "data_warm,data_hot"
              }
            }
          },
          "hidden": "true",
          "number_of_shards": "1",
          "provided_name": ".ds-test_ilm_index-2023.01.20-000001",
          "default_pipeline": "ip_timestamp",
          "creation_date": "1674200701723",
          "priority": "50",
          "number_of_replicas": "0",
          "uuid": "Glg3Wb_cThWRVexefxq1QA",
          "version": {
            "created": "8050399"
          }
        }
      },
      "data_stream": "test_ilm_index"
    },
    ".ds-test_ilm_index-2023.01.20-000002": {
      "aliases": {},
      "mappings": {
        "_data_stream_timestamp": {
          "enabled": true
        },
        "properties": {
          "@timestamp": {
            "type": "date"
          },
          "test": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          }
        }
      },
      "settings": {
        "index": {
          "lifecycle": {
            "name": "ilm_test_activities",
            "indexing_complete": "true"
          },
          "routing": {
            "allocation": {
              "include": {
                "_tier_preference": "data_warm,data_hot"
              }
            }
          },
          "hidden": "true",
          "number_of_shards": "1",
          "provided_name": ".ds-test_ilm_index-2023.01.20-000002",
          "default_pipeline": "ip_timestamp",
          "creation_date": "1674204674509",
          "priority": "50",
          "number_of_replicas": "0",
          "uuid": "WdCkeUBMQFW1jJNaZBiutg",
          "version": {
            "created": "8050399"
          }
        }
      },
      "data_stream": "test_ilm_index"
    },
    ".ds-test_ilm_index-2023.01.20-000003": {
      "aliases": {},
      "mappings": {
        "_data_stream_timestamp": {
          "enabled": true
        },
        "properties": {
          "@timestamp": {
            "type": "date"
          },
          "test": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          }
        }
      },
      "settings": {
        "index": {
          "lifecycle": {
            "name": "ilm_test_activities",
            "indexing_complete": "true"
          },
          "routing": {
            "allocation": {
              "include": {
                "_tier_preference": "data_warm,data_hot"
              }
            }
          },
          "hidden": "true",
          "number_of_shards": "1",
          "provided_name": ".ds-test_ilm_index-2023.01.20-000003",
          "default_pipeline": "ip_timestamp",
          "creation_date": "1674208874425",
          "priority": "50",
          "number_of_replicas": "0",
          "uuid": "REzMYu_oQLyGNk805Wwnyg",
          "version": {
            "created": "8050399"
          }
        }
      },
      "data_stream": "test_ilm_index"
    },
    ".ds-test_ilm_index-2023.01.20-000004": {
      "aliases": {},
      "mappings": {
        "_data_stream_timestamp": {
          "enabled": true
        },
        "properties": {
          "@timestamp": {
            "type": "date"
          },
          "test": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          }
        }
      },
      "settings": {
        "index": {
          "lifecycle": {
            "name": "ilm_test_activities",
            "indexing_complete": "true"
          },
          "routing": {
            "allocation": {
              "include": {
                "_tier_preference": "data_warm,data_hot"
              }
            }
          },
          "hidden": "true",
          "number_of_shards": "1",
          "provided_name": ".ds-test_ilm_index-2023.01.20-000004",
          "default_pipeline": "ip_timestamp",
          "creation_date": "1674213077692",
          "priority": "50",
          "number_of_replicas": "0",
          "uuid": "GQY2-a2kS3aPkrdWV3vsdw",
          "version": {
            "created": "8050399"
          }
        }
      },
      "data_stream": "test_ilm_index"
    },
    ".ds-test_ilm_index-2023.01.20-000005": {
      "aliases": {},
      "mappings": {
        "_data_stream_timestamp": {
          "enabled": true
        },
        "properties": {
          "@timestamp": {
            "type": "date"
          },
          "test": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          }
        }
      },
      "settings": {
        "index": {
          "lifecycle": {
            "name": "ilm_test_activities",
            "indexing_complete": "true"
          },
          "routing": {
            "allocation": {
              "include": {
                "_tier_preference": "data_warm,data_hot"
              }
            }
          },
          "hidden": "true",
          "number_of_shards": "1",
          "provided_name": ".ds-test_ilm_index-2023.01.20-000005",
          "default_pipeline": "ip_timestamp",
          "creation_date": "1674219681199",
          "priority": "50",
          "number_of_replicas": "0",
          "uuid": "DH_O-JDASNyz94nRtj-e7A",
          "version": {
            "created": "8050399"
          }
        }
      },
      "data_stream": "test_ilm_index"
    },
    ".ds-test_ilm_index-2023.01.20-000006": {
      "aliases": {},
      "mappings": {
        "_data_stream_timestamp": {
          "enabled": true
        },
        "properties": {
          "@timestamp": {
            "type": "date"
          },
          "test": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          }
        }
      },
      "settings": {
        "index": {
          "lifecycle": {
            "name": "ilm_test_activities"
          },
          "routing": {
            "allocation": {
              "include": {
                "_tier_preference": "data_hot"
              }
            }
          },
          "hidden": "true",
          "number_of_shards": "1",
          "provided_name": ".ds-test_ilm_index-2023.01.20-000006",
          "default_pipeline": "ip_timestamp",
          "creation_date": "1674225077571",
          "priority": "100",
          "number_of_replicas": "0",
          "uuid": "ndl6TE29TKO3H6lEfM6Rrw",
          "version": {
            "created": "8050399"
          }
        }
      },
      "data_stream": "test_ilm_index"
    }
  }

We have updated the test environment:
We have added a cluster level ilm setting to allow index rollover with 0 documents.

indices.lifecycle.rollover.only_if_has_documents= false

And stopped the cron job pushing data every 5 minutes

The Datastream is now rolling index with expected behaviour.

What could be the reason behind this? what could be the issue?