Datastream under a policy behaving unstable

[Mubolio]

Hello,

I have a datastream that should keep data only for the latest 15 days, documents that are older than 15 days will be deleted(based on the @timestamp field).

This is the index template attached to the datastream:

{
  "template": {
    "settings": {
      "index": {
        "lifecycle": {
          "name": "policy_name"
        },
        "look_ahead_time": "7d",
        "codec": "best_compression",
        "routing": {
          "allocation": {
            "include": {
              "_tier_preference": "data_hot"
            }
          }
        },
        "number_of_replicas": "0"
      }
    },
    "mappings": {
      "properties": {
        "date_update": {
          "type": "date",
          "format": "strict_date_optional_time"
        },
        "geo_bbox": {
          "properties": {
            "location": {
              "type": "geo_shape"
            }
          }
        },
        "geo_point": {
          "properties": {
            "location": {
              "type": "geo_point"
            }
          }
        }
      }
    },
    "aliases": {}
  }
}

This is the data policy attached to the index template:

PUT _ilm/policy/policy_name
{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {
          "set_priority": {
            "priority": 100
          }
        }
      },
      "delete": {
        "min_age": "15d",
        "actions": {
          "delete": {
            "delete_searchable_snapshot": true
          }
        }
      }
    }
  }
}

I can see that the time window for the stored documents in the data stream is variable/unstable is not always 15 days, what I expect and I would like is that it just keeps docs for the latest 15 days, that window should be moving as the time lapses. (I think the time windows increases till has 15 days and then closes up to 0 days, and it fill again but I'm not 100% sure).

Please, any help would be very much appreciated.

Best regards.

[Mubolio]

Hello,

this was a mistake from my side since I assumed that the ILM was being applied at the level of document and no of index.

So the solution should be to do a rollover in the cold phase, generating a new index each day and switch to the delete phase when the indices are older than 7 days, in this case.

Regards.