ILM Hot, Warm, Cold not moving indexes

[DaddyYusk (Jonathan VINCENT)]

Hi,

Despite the ILM policy applied to all logs (Managed), the indexes are moved to the Warm node but not deleted on the Hot node and I'm actually reaching disk capacity on the Hot node...

Here is my Elastic Cluster :

w         -      ovh-dataw-1
hist      -      ovh-datah-1
mr        -      ovh-master-2
mr        -      ovh-master-3
mr        *      ovh-master-1
c         -      ovh-datac-1

Here is the output of a problematic index :

index                                       shard prirep state       docs store dataset ip            node
.ds-logs-winlog.winlog-dc-2023.12.06-000001 0     p      STARTED 57415011  50gb    50gb 192.168.74.42 ovh-datah-1
.ds-logs-winlog.winlog-dc-2023.12.06-000001 0     r      STARTED 57415011  50gb    50gb 192.168.74.40 ovh-dataw-1

Running this query :

GET /.ds-logs-winlog.winlog-dc-2023.12.06-000001/_ilm/explain?human

I get this interesting output :

"message": "[.ds-logs-winlog.winlog-dc-2023.12.06-000001] lifecycle action [migrate] waiting for [1] shards to be moved to the [data_warm] tier (tier migration preference configuration is [data_warm, data_hot])",

Any idea please?
Regards.

[leandrojmp (Leandro Pereira)]

@DaddyYusk

Any idea please?

In the ILM policy you configured it to remove the replicas?

I'm not sure, but I think this can impact as you only have one warm node.

Try to remove the replica of this index and see if it will move out from the hot node.

PUT /.ds-logs-winlog.winlog-dc-2023.12.06-000001/_settings
{
"index" : {
  "number_of_replicas" : 0
}
}

[DaddyYusk (Jonathan VINCENT)]

Hi @leandrojmp,

And thanks a lot for your help!
Indeed, it solved my issue.

@leandrojmp

In the ILM policy you configured it to remove the replicas?

How to configure that in the ILM (managed) please?

Now I need to configure all my indexes with no replica. So I will proceed like that :

PUT /_all/_settings
{
  "index" : {
      "number_of_replicas" : 0
  }
}

And for new indexes, as I'm using a Managed Index Template but there are many... I'm not sure how to proceed.
Should I need to edit only "logs" or each "logs-elastic_agent.XXX"?

Thanks again!

[DaddyYusk (Jonathan VINCENT)]

Finally, for setting replica to 0 for new indexes, I go for editing all index templates manually.
That was tedious but doable in less than 10min.
Thanks again for your help!