Greetings!
We are in the process of integrating Elastic Observability into our system. At present, we operate a variety of unique services. Each of these services transmits telemetry data first to an OTEL collector and then to the Elastic APM server. The services are identified by individual names, leading to a corresponding number of data streams, such as logs-apm.app.1234-default and logs-apm.app.5678-default, where numbers like 1234 and 5678 denote the names of our services.
Our main aim is to identify an efficient strategy for setting up specific ILM (Information Lifecycle Management) policies for each distinct data stream that carries our telemetry data. The intention is to allow for tailored storage times for the telemetry data of specific services.
We would appreciate your guidance on the optimal setup for components such as Index Lifecycle Policies and Index Templates to meet our requirements.
A point for consideration: is it advisable to clone the default logs-apm.app index template, create a distinct index pattern for the needed data stream, and apply a dedicated ILM policy to this new template? Or might there be a more streamlined solution? We're also contemplating the implications of this approach, such as potential complications in future Elastic updates or modifications to the current default logs-apm.app index template that oversees all log data streams. Your insights on this matter would be greatly appreciated.
Thank you in advance!