Data streams vs ILM when using Fleet

[djkprojects]

Hello,

We are moving away from Beats towards the Centralised Elastic Agent / Fleet but struggling to understand the concept behind how the new approach is meant to work with ILM. We have multiple infrastructures to pull metrics from and we use namespace to differentiate between them e.g.:

Infrastructure 1
metrics-system.cpu-infra1
metrics-system.metrics-infra1
... and so on

Infrastructure 2
metrics-system.cpu-infra2
metrics-system.metrics-infra2
... and so on

By default they all use the built-in metrics ILM but we want to have different policies for Infra1 and Infra2.

We can create new templates and apply new ILMs but that would mean creating new template for each data stream which would make it really difficult to maintain with many infrastructures.

We are trying to understand the correct approach here is.

Thanks

[leandrojmp (Leandro Pereira)]

@djkprojects

We are trying to understand the correct approach here is.

Unfortunately what you said is already the recommended approach by elastic, it is documented here.

You would need to to create a custom template for every dataset and every namespace, which is something really hard to maintain.

I had the same issue when I started to use the integrations to get some logs and by suggestion of someone from Elastic I opened this issue on Github proposing some changes.

Iif you can wait I would suggest that you keep using Beats instead of Elastic Agent, in my experience Elastic Agent makes it easier to get the data, but make the management of the indices twice or more hard.

[djkprojects]

We've been very reluctant to switch over to Elastic Agent because of that and other issues we've encountered but the decision has already been made.

What we were thinking of is rather than creating a copy of a template for each data stream we would create a combined template which would contain all the individual components e.g.:

and then apply ILM to the template.

but I'm not sure if this is not going to cause mapping conflicts

Any ideas if this is an option?

Thanks

[leandrojmp (Leandro Pereira)]

I'm not sure, but I think that this approach may lead to some mapping conflicts and maybe also need to be redone when you update an agent.

Customizing anything related to mappings and ingest pipelines with the Elastic Agent gives you a lot of work, there are some work being done to help this, but at this moment I would avoid doing that.