To big index for ML job - how to add ILM?

Elastic Stack Elasticsearch
1

Hello Everyone,

I’ve encountered an issue while working with Machine Learning jobs in Elasticsearch, where the result indices aren't being properly managed post-rollover.
I have to figured out something because index for this specific ML job was above 600 GB.
I need any ILM and rollover strategy to manage this inidices.

Problem Description:
I started by creating an ML job in Elasticsearch, specifying the results_index_name as a concrete index name ending with a sequence number (e.g., custom-ml-anomalies-metrics-count-high_mean-v12-000001). Everything worked fine until the index met the rollover conditions set in the ILM policy. The index was rolled over to a new index (e.g., ...-000002), but the ML job continued writing to the old index, resulting in new data not being recorded in the newly created index.

Steps Taken:

  1. I attempted to manually rollover the index using the Rollover API, but after that new custom-ml-anomalies-metrics-count-high_mean-v12-000002 was not "assigned" to ML job.
    ML job stop adding a documents to `custom-ml-anomalies-metrics-count-high_mean-v12-000001' and 'custom-ml-anomalies-metrics-count-high_mean-v12-000002'
  2. I created an alias for the result index with the is_write_index flag set to true, hoping this would solve the issue of updating the target index in the ML job.

Configuration:

  • I set an alias for the result index (e.g., ml-results-custom-metrics-v12) as results_index_name in the ML job.
  • I configured an ILM policy to use this alias as the rollover_alias.

Questions:
Do you have any suggestions on how yo achieve my goal?

I would appreciate any comments and suggestions.

I followed by this ROTATE ML INDEXES but without luck.