Hi There,
I’m a newbie to Elastic and looking to solve an issue that I am having trouble with.
Essentially I am looking to set up a Telemetry stack across various apps we maintain, and am trialing an approach for a single app currently.
I currently have instrumented this app with OpenTelemetry (logs, traces and metrics) and am sending this data to an OpenTelemetry collector. The collector then forwards the data onto an APM server, which is then processed and passed on to Elastic. Currently this is all happening in a local instance, before I look to set this up using our Elastic Cloud account.
I am currently focused on logs, which is being used to capture basic performance metrics (web vitals), analytics data and error logs. I am looking to achieve different data retention policies from our analytics logs versus error logs (i.e. analytics data we keep for longer-term, error data shorter). I have had a look at some existing posts and tried to find something which aligns to this goal, but have not been successful.
I have experimented with ways to possibly achieve this, and so far have been able to modify the default ingest pipelines in order to reroute the logs into different data streams based on the log level (so currently have INFO → logs-apm.app.<app-name>-analytics and ERROR → logs-apm.app.<app-name>-errors)
However what I cannot figure out is how to set up different ILM policies for these. I have looked up setting up custom Index Templates based off the original logs-apm.app@template with differing index patterns that match these stream names, but I don’t think this is achievable based on my understanding of how the APM data is processed, since I see that the original Index Template is associated with these rerouted data streams regardless (and therefore inherits the associated ILM policy).
I’m quite stuck. I’ve also tried to handle this on the OpenTelemetry side (to try and route the different log levels into different streams) but I cant see any configurations on the APM server or OTLP Elastic Exporter to be able to do this.
Any thoughts, ideas or tips would be appreciated!