I'm starting to use the Fleet Server and the Elastic Agent to be able to use some integrations and because of that I'm also using data streams for the first time.
I have a fleet server and one agent running where I configured two simple integrations, the AbuseCH for Threatintel and the Cisco Duo integration.
When I looked in the datastreams, I saw that every integration uses the same index lifecycle policy, which would be impossible to use since some integrations may create 1 GB per month and other 200 GB per day, so there is the need to have different lifecycle policies for different integrations.
But I did not find yet an easy way to change the retention, the documentation has this page on how to customize the retention for managed data streams, but this seems a lot of work for something that should be easier, for just the 2 integrations I mentioned I would need to change at least 6 or 7 data streams.
Is there a way to set a custom ILM policy while adding a new integration? Is this feature already requested?
I know this documentation, it is the same one I've linked in the post, that's why I've asked if there is another way.
As I said, with just the 2 integrations I'm using I would need to repeat the steps in the documentation at least 6 times, this is a lot of work for something that should be an option while creating an integration.
It makes no sense to have the same lifecycle policy being applied to data that may have a huge difference in volumetry.
Elastic Agent integrations should have the option to set a custom lifecycle policy while creating the integration.
I'm testing if I can create a simple component template and apply this same component template to the integrations to make things less complicated.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.