Configure global retention time (ILM) for all logs and metrics

Hello!

I'm using Elastic Agent with Fleet and Integrations. I'd like all collected data (metrics and logs) to be deleted after about N days. Is there a more elegant solution to this other than modifying the default "managed" logs and metrics lifecycle policies?

The method documented here is waaay too granular, I don't know what integrations will be enabled in the future.

Looking at how the shipped index templates include component templates, I don't think creating a logs@custom component would work.

Could there be any side-effects of editing the aforementioned managed policies? Also, would a stack update override my changes?

Hi @lpeter

What Version are you on...

What are you using for shipping telemetry Beats / Agent / Both?

Edit: yup not the right place

Thanks for your input.

Yeah, sorry, forgot to include the version: this is a new system still under planning, so we can assume the latest version for now (8.12.1).

I'd like to only use the Agent if possible, less variety is preferred. Also the docs are giving me the vibe that that's the way forward for new systems. :slight_smile:

Regarding logs@custom: As far as I can tell it's only included by the generic logs index template, but integrations (usually) have a more specific template (like logs-auditd_manager.auditd) with higher priority, and those don't seem to include logs@custom. But I'll actually test it tomorrow, maybe it doesn't work the way I think it does.

(Looking around I see there's now a simpler lifecycle API for data streams in tech preview. Cool.)

Yes but it will be very simple...

You are right. The logs@custom is only used by default... probably not the right place... I will look around

The policy gets set globally in logs@settings

And, of course, you can actually edit the default "Managed" logs ILM policy .. Kibana will yell at you, but you absolutely can... you will just need to be careful when updating, I changed mine long ago and have not had any issues...

It has not been overwritten (but I check after every upgrade)

1 Like

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.