I recently migrated from beats to fleet managed agents (about 500 servers total, mostly windows) and am wondering what best practice is for managing the system managed data streams for logs and metrics (logs-system*, metrics-system*).
With beats, all my data was being indexed into a single data stream. Now, with the agent, there’s several data streams - for each telemetry type and their subtype.
My predicament is that these new managed data streams can take awhile small depending on the data it’s collecting. For instance, the metrics-uptime data stream only ingests about 500mb every 6 months. The metrics-process data stream ingests about 100gb per month (we’re collecting metrics for a bunch of custom processes). The other data stream ingest at a different rate as what I mentioned above.
What would make sense as far as ILM goes if I want to retain data for a minimum of a year but I want as few ILMs to manage as possible? Is it possible to set a minimum shard or index size before rollover?