ILM Setup support for ELK 8.3

[shailendra1 (shailendra sharma)]

Dear All,

i am looking some suggestions for setting up the ILM for my data/streaming/beats . i have below senerios to accomplished and keep running my platforms.

1. Everyday the indices are generating with abc.yyy.mm.dd format, without any data size limits of indices within the logstash mapping and patterns, the problem here how can i setup my ILM to avoid the flood of data sizes (regular pattern is around 1gb data but suddenly if data comes to 50gb)

2. can we setup ILM policy or rollover configuration in the Logstash pipelines with a indices size limits?

3. is there any better approach to keep my platform up and running with limited to the disk spaces/cluster storages.

4. i am using the ELK 8.3 with FB agents in multiple hosts.

Thank you in advance for your early responses and suggestions.

[sholzhauer]

it sounds like you are looking for the ilm rollover action based on index size:

Rollover | Elasticsearch Guide [8.3] | Elastic

This would rollover (create a new index) when the current reaches the configured size (checks every 10 min).

e.g:

{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover" : {
            "max_size": "5gb"
          }
        }
      },
      "delete": {
        "min_age": "2h"
      }
    }
  }
}

This policy would create indices of roughly 5gb and after rollover keep your data for an additional 2h.