Need Explanation ILM / DSL / closing indice and stuff

Elastic Stack Elasticsearch
1

Hi ,

Hi Everyone ,

I'm trying to understand what I've done with ILM and it looks messy :smiley:

I need explanation and some help on good practices for that.

The Setup :

I have an Elasticsearch cluster with 10 Data nodes. 250Gb storage per node. Version 8.18.1

I have multiple filebeats sending datas to Elasticsearch ( Around 10 filebeats ). I'll take one for my example with is : analytics-kubernetes-apps-np

Here is the filebeat configuration

                        output.elasticsearch:
                          hosts: masked
                          ssl.verification_mode : "none"
                          username: masked
                          password: "masked"
                          pipeline: kubernetes-nginx-routing
                          index: "analytics-kubernetes-apps-np-%{+yyyy.MM.dd}"
                        setup.ilm.enabled: true
                        setup.ilm.policy_name: "analytics-kubernetes-apps-np"
                        setup.ilm.rollover_alias: "analytics-kubernetes-apps-np"
                        setup.template.name: "analytics-kubernetes-apps-np"
                        setup.template.pattern: "analytics-kubernetes-apps-np-*"

So When I Create the filebeat, It load my index template and Create a datastream.

Here is the Lifecycle Policy

PUT _ilm/policy/analytics-kubernetes-apps-np
{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {
          "rollover": {
            "max_age": "1d"
          }
        }
      },
      "delete": {
        "min_age": "3d",
        "actions": {
          "delete": {
            "delete_searchable_snapshot": true
          }
        }
      }
    }
  }
}

So everything goes fine at this point. My ilm rollover my index and create indice every days. Data is cleaned up every 3 days.

The problem :

This creates a lot of datasources and it is fine. Here are datasources

https://ibb.co/dsrqDKtQ

And this also create a lot of indices

https://ibb.co/d47p7JWv

I understand that theses indices are all rollover indices. But I don't understand why they are still here. There are no documents in here . And the indice is still active and open !

image
image1676×646 59.5 KB

https://ibb.co/QvLR5k7P

The issue here is that the indice is still here. And it initialize shards !! So at one point I have too many shards and my cluster can't initialize more shards. It switch then to unhealthy.

I have to manually delete thoses old indices to remove some shard and make it healthy again.

Is there something I'm doing wrong ?

Thanks a lot for your support.

Regards

Sorry for the links , cant post more that one image in the post :confused: