Elasticsearch ILM Policies

Elastic Stack Elasticsearch
1

Hi All

Currently we are sending logs to elasticsearch and it's configured in such a way that daily datastreams are created. For Eg: logs-app1-2023-09-23, logs-app1-2023-09-24 etc. Goal is to keep logs for 3 days. We have created an ILM policy as follows:

{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {
          "rollover": {
            "max_primary_shard_size": "50gb",
            "max_age": "1d"
          },
          "set_priority": {
            "priority": 100
          }
        }
      },
      "warm": {
        "min_age": "1d",
        "actions": {
          "set_priority": {
            "priority": 50
          }
        }
      },
      "delete": {
        "min_age": "3d",
        "actions": {
          "delete": {
            "delete_searchable_snapshot": true
          }
        }
      }
    }
  }
}

Wanted to some clarification here. So let's say my data-stream was created - logs-app1-2023-09-23, so there will be a corresonding index created on 23th September. On 24th the index will be rolled over and a new index gets created. Now since I have set the min age as 1 day in the warm phase after 1 day i.e is on 25th Sept the rolled over index will be moved to warm phase and then stay there for 2 days and get's deleted on 27th September (3 days from rollover).

My doubt here is once the index is rolled over and new index get's created there is going to be no data written to that new index since I have daily datastreams. Does elasticsearch provide a way to delete streams on a periodic basis too?

Some guidance for the best approach to be taken here would help.

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.