Is there an easy way to change the Lifecycle Policy of a Fleet managed data stream?


I'm starting to use the Fleet Server and the Elastic Agent to be able to use some integrations and because of that I'm also using data streams for the first time.

I have a fleet server and one agent running where I configured two simple integrations, the AbuseCH for Threatintel and the Cisco Duo integration.

When I looked in the datastreams, I saw that every integration uses the same index lifecycle policy, which would be impossible to use since some integrations may create 1 GB per month and other 200 GB per day, so there is the need to have different lifecycle policies for different integrations.

But I did not find yet an easy way to change the retention, the documentation has this page on how to customize the retention for managed data streams, but this seems a lot of work for something that should be easier, for just the 2 integrations I mentioned I would need to change at least 6 or 7 data streams.

Is there a way to set a custom ILM policy while adding a new integration? Is this feature already requested?

1 Like

Hi @leandrojmp

Yes it's possible to change the lifecycle policy of a datastream managed by Fleet see the doc here for more info Tutorial: Customize data retention policies | Fleet and Elastic Agent Guide [8.4] | Elastic

Hello @nchaulet,

I know this documentation, it is the same one I've linked in the post, that's why I've asked if there is another way.

As I said, with just the 2 integrations I'm using I would need to repeat the steps in the documentation at least 6 times, this is a lot of work for something that should be an option while creating an integration.

It makes no sense to have the same lifecycle policy being applied to data that may have a huge difference in volumetry.

Elastic Agent integrations should have the option to set a custom lifecycle policy while creating the integration.

I'm testing if I can create a simple component template and apply this same component template to the integrations to make things less complicated.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.