Rollups and ILM

[RichardH (Richard Hepplestone)]

Hi All,

A bit of a newbie question.

Am I correct in my understanding that Rollups do not delete any data. They just create new aggregate/summary indexes. I would need to set up ILM to delete old granular data from the original indices after rollup?

Then, when configuring rollups. If I set the latency buffer to 7d does that mean it only performs rollups on data older than 7 days?

The time bucket size under Date Histogram - does this set the granularity of the rolled up data. ie. if set to 60m the rolled up data will include 1 hour averages of the metrics selected. Is this correct?

Or should I be working with the Histogram interval width setting to control this? (or is it both)?

Metrics selects the metrics I want to collect.

So if (using the Rollups configuration in Kibana) I select:

   "Latency buffer (optional)" = 7d.
   "Time Bucket" size = 60m
    A list of Monitoring metrics under "Histogram" (e.g. beats_stats.metrics.apm- 
          server.decoder.deflate.content-length) with a Histogram interval of 100
    The same Monitoring metrics under "Metrics" (e.g. beats_stats.metrics.apm- 
           server.decoder.deflate.content-length) and select the "Average" check box

Would it then rollup all Averages in the chosen metrics that are older than 7d into 60min buckets?

I could then (assuming I don't want it anymore, set up the ILM policy on the original indexes to delete after 7 days.

Thanks

[spinscale (Alexander Reelsen)]

regarding integration with index lifecycle management, you may want to follow this issue on github: https://github.com/elastic/elasticsearch/issues/48003

[birko]

Hi Alexander,
so nowadays it's not possible to manage old rollup data with ILM? Do you have any proposal how to manage it without ILM effectively, please?

Thanks