Inconsistency in data stream rollover

We want hourly rollover of the index and then index should get deleted after 12 hour.

To achieve this we have created data stream with a ILM policy which rollover index every 1hr and then the index goes into the warm phase and after 12 hour index goes into deleted phase.

Setup:
- Elasticsearch 8.6 docker on ubuntu 22.4
- We have setup a test cron job which push data to the index in every 5th min.
- Below are the scripts for the ILM policy and index template

	PUT _ilm/policy/ilm_test_activities2
	{
		"policy": {
			"phases": {
				"hot": {
					"min_age": "0s",
					"actions": {
						"set_priority": {
							"priority": 100
						},
						"rollover": {             
							"max_age": "1h"            
						},
					}
				},
				"warm": {
					"min_age": "0s",
					"actions": {
						"set_priority": {
							"priority": 50
						}
					}
				},
				"delete": {
					"min_age": "12h",
					"actions": {
					"delete": {}
					}
				}
			}
		}
	}
		
	PUT /_index_template/it_test_activities2
	{
		"index_patterns" : ["test-activity2-*"],
		"priority" : 1,
		"data_stream": { },
		"template": {
			"settings" : {
				"index" : {
					"lifecycle" : {
						"name" : "ilm_test_activities2"
					},
					"default_pipeline": "ip_timestamp",
					"number_of_replicas": 0
				}
			}
		}
	}

Expected behaviour for index rollover:
-Every hour new index should get created.

Current behaviour:
Index is not getting rollover every hour.
Test result:

  • First index created at: 07:45
  • Second index rollover : 08:51 // after 1 hr 06 mins
  • Third index rolleover : 10:01 // after 1 hr 10 mins
  • Fourth index rolleover: 11:11 // after 1 hr 10 mins
  • Fifth index rolleover : 13:01 // after 1 hr 50 mins
  • sixth index rolleover : 14:31 // after 1 hr 30 mins

We have observed more time gap(upto 3-5 hr) in the index rollover.

Is there any misconfiguration from our end?

Data stream information:

{
    ".ds-test_ilm_index-2023.01.20-000001": {
      "aliases": {},
      "mappings": {
        "_data_stream_timestamp": {
          "enabled": true
        },
        "properties": {
          "@timestamp": {
            "type": "date"
          },
          "test": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          }
        }
      },
      "settings": {
        "index": {
          "lifecycle": {
            "name": "ilm_test_activities",
            "indexing_complete": "true"
          },
          "routing": {
            "allocation": {
              "include": {
                "_tier_preference": "data_warm,data_hot"
              }
            }
          },
          "hidden": "true",
          "number_of_shards": "1",
          "provided_name": ".ds-test_ilm_index-2023.01.20-000001",
          "default_pipeline": "ip_timestamp",
          "creation_date": "1674200701723",
          "priority": "50",
          "number_of_replicas": "0",
          "uuid": "Glg3Wb_cThWRVexefxq1QA",
          "version": {
            "created": "8050399"
          }
        }
      },
      "data_stream": "test_ilm_index"
    },
    ".ds-test_ilm_index-2023.01.20-000002": {
      "aliases": {},
      "mappings": {
        "_data_stream_timestamp": {
          "enabled": true
        },
        "properties": {
          "@timestamp": {
            "type": "date"
          },
          "test": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          }
        }
      },
      "settings": {
        "index": {
          "lifecycle": {
            "name": "ilm_test_activities",
            "indexing_complete": "true"
          },
          "routing": {
            "allocation": {
              "include": {
                "_tier_preference": "data_warm,data_hot"
              }
            }
          },
          "hidden": "true",
          "number_of_shards": "1",
          "provided_name": ".ds-test_ilm_index-2023.01.20-000002",
          "default_pipeline": "ip_timestamp",
          "creation_date": "1674204674509",
          "priority": "50",
          "number_of_replicas": "0",
          "uuid": "WdCkeUBMQFW1jJNaZBiutg",
          "version": {
            "created": "8050399"
          }
        }
      },
      "data_stream": "test_ilm_index"
    },
    ".ds-test_ilm_index-2023.01.20-000003": {
      "aliases": {},
      "mappings": {
        "_data_stream_timestamp": {
          "enabled": true
        },
        "properties": {
          "@timestamp": {
            "type": "date"
          },
          "test": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          }
        }
      },
      "settings": {
        "index": {
          "lifecycle": {
            "name": "ilm_test_activities",
            "indexing_complete": "true"
          },
          "routing": {
            "allocation": {
              "include": {
                "_tier_preference": "data_warm,data_hot"
              }
            }
          },
          "hidden": "true",
          "number_of_shards": "1",
          "provided_name": ".ds-test_ilm_index-2023.01.20-000003",
          "default_pipeline": "ip_timestamp",
          "creation_date": "1674208874425",
          "priority": "50",
          "number_of_replicas": "0",
          "uuid": "REzMYu_oQLyGNk805Wwnyg",
          "version": {
            "created": "8050399"
          }
        }
      },
      "data_stream": "test_ilm_index"
    },
    ".ds-test_ilm_index-2023.01.20-000004": {
      "aliases": {},
      "mappings": {
        "_data_stream_timestamp": {
          "enabled": true
        },
        "properties": {
          "@timestamp": {
            "type": "date"
          },
          "test": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          }
        }
      },
      "settings": {
        "index": {
          "lifecycle": {
            "name": "ilm_test_activities",
            "indexing_complete": "true"
          },
          "routing": {
            "allocation": {
              "include": {
                "_tier_preference": "data_warm,data_hot"
              }
            }
          },
          "hidden": "true",
          "number_of_shards": "1",
          "provided_name": ".ds-test_ilm_index-2023.01.20-000004",
          "default_pipeline": "ip_timestamp",
          "creation_date": "1674213077692",
          "priority": "50",
          "number_of_replicas": "0",
          "uuid": "GQY2-a2kS3aPkrdWV3vsdw",
          "version": {
            "created": "8050399"
          }
        }
      },
      "data_stream": "test_ilm_index"
    },
    ".ds-test_ilm_index-2023.01.20-000005": {
      "aliases": {},
      "mappings": {
        "_data_stream_timestamp": {
          "enabled": true
        },
        "properties": {
          "@timestamp": {
            "type": "date"
          },
          "test": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          }
        }
      },
      "settings": {
        "index": {
          "lifecycle": {
            "name": "ilm_test_activities",
            "indexing_complete": "true"
          },
          "routing": {
            "allocation": {
              "include": {
                "_tier_preference": "data_warm,data_hot"
              }
            }
          },
          "hidden": "true",
          "number_of_shards": "1",
          "provided_name": ".ds-test_ilm_index-2023.01.20-000005",
          "default_pipeline": "ip_timestamp",
          "creation_date": "1674219681199",
          "priority": "50",
          "number_of_replicas": "0",
          "uuid": "DH_O-JDASNyz94nRtj-e7A",
          "version": {
            "created": "8050399"
          }
        }
      },
      "data_stream": "test_ilm_index"
    },
    ".ds-test_ilm_index-2023.01.20-000006": {
      "aliases": {},
      "mappings": {
        "_data_stream_timestamp": {
          "enabled": true
        },
        "properties": {
          "@timestamp": {
            "type": "date"
          },
          "test": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          }
        }
      },
      "settings": {
        "index": {
          "lifecycle": {
            "name": "ilm_test_activities"
          },
          "routing": {
            "allocation": {
              "include": {
                "_tier_preference": "data_hot"
              }
            }
          },
          "hidden": "true",
          "number_of_shards": "1",
          "provided_name": ".ds-test_ilm_index-2023.01.20-000006",
          "default_pipeline": "ip_timestamp",
          "creation_date": "1674225077571",
          "priority": "100",
          "number_of_replicas": "0",
          "uuid": "ndl6TE29TKO3H6lEfM6Rrw",
          "version": {
            "created": "8050399"
          }
        }
      },
      "data_stream": "test_ilm_index"
    }
  }

We have updated the test environment:
We have added a cluster level ilm setting to allow index rollover with 0 documents.

indices.lifecycle.rollover.only_if_has_documents= false

And stopped the cron job pushing data every 5 minutes

The Datastream is now rolling index with expected behaviour.

What could be the reason behind this? what could be the issue?

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.