1h difference between rollover time and action time

Elastic Stack Elasticsearch
1

Hi,

I noticed that despite my indexes are being rolled over at midnight, the "index phase time" and "index action time" are with 2h difference apart, like:

{
  "indices" : {
    "suricata-ids-2022.03.28-000021" : {
      "index" : "suricata-ids-2022.03.28-000021",
      "managed" : true,
      "policy" : "filebeat",
      "lifecycle_date_millis" : 1648425600000,
      "age" : "21.81h",
      "phase" : "hot",
      "phase_time_millis" : 1648433109374,
      "action" : "rollover",
      "action_time_millis" : 1648433109733,
      "step" : "check-rollover-ready",
      "step_time_millis" : 1648433109733,
      "phase_execution" : {
        "policy" : "filebeat",
        "phase_definition" : {
          "min_age" : "0ms",
          "actions" : {
            "rollover" : {
              "max_age" : "1d"
            }
          }
        },
        "version" : 50,
        "modified_date_in_millis" : 1648501759739
      }
    }
  }
}

So, index suricata-ids-2022.03.28-000021 only starts to have data from 2AM forward. The previous data from 0AM to 2AM are still being indexed by suricata-ids-2022.03.27-000021

This is my ILM policy:

PUT _ilm/policy/filebeat
{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {
          "rollover": {
            "max_age": "1d"
          }
        }
      },
      "warm": {
        "min_age": "5d",
        "actions": {
          "set_priority": {
            "priority": 50
          }
        }
      },
      "delete": {
        "min_age": "10d",
        "actions": {
          "delete": {
            "delete_searchable_snapshot": true
          }
        }
      }
    }
  }
}

I am using the "parse_origination_date": "true" option on my index template. What i want is to rollover the index at midnight and immediately start writing the data into the index.

I am using ELK stack 7.13.4

2

Rollover timing is based on the index creation time, which may not have happened immediately at midnight to start with, plus any gap that might happen between each policy run, as ILM only checks every 10 minutes.

So it's unlikely that it rollover exactly at midnight.

3

Anyway, is it normal/plausible that exists a 2 hour gap?

4

Is it always 2 hours difference?

If the difference is always the same or very close, maybe it is a time zone issue?

In which time zone are you?

5

Taking into account the 29/03/2022 index:

{
  "indices" : {
    "suricata-ids-2022.03.29-000022" : {
      "index" : "suricata-ids-2022.03.29-000022",
      "managed" : true,
      "policy" : "filebeat",
      "lifecycle_date_millis" : 1648512000000,
      "age" : "9.05h",
      "phase" : "hot",
      "phase_time_millis" : 1648519755243,
      "action" : "rollover",
      "action_time_millis" : 1648519755640,
      "step" : "check-rollover-ready",
      "step_time_millis" : 1648519755640,
      "phase_execution" : {
        "policy" : "filebeat",
        "phase_definition" : {
          "min_age" : "0ms",
          "actions" : {
            "rollover" : {
              "max_age" : "1d"
            }
          }
        },
        "version" : 50,
        "modified_date_in_millis" : 1648501759739
      }
    }
  }
}

The index is created at 00:00 but the phase time is only around 2AM. The 2 hour gap as been regular. My timezone is Europe/Lisbon.

6

Still happening today.

Capture
Capture896×569 47.7 KB

The index is created at midnight but only starts to get data from 2AM forward (3AM in my local time)

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.