Hello!
Has anyone tried to rollup APM indices? Is it even possible?
Thank you in advance for any feedback 
Hello @A_B,
can I ask about your use case for rollups with APM?
In case you are only looking for a way to manage data retention, you can make use of the index lifecycle management, which is enabled by default. This allows to keep more fine granular data (e.g. spans) or data that are outdated faster (e.g. errors) for a shorter period than high level data (e.g. transactions). APM Server sets up indices with the same retention period by default, but the index lifecycle policies are configurable. Read more about how to configure ILM policies for your data .
Hi @simitt,
thanks for the suggestions. Ideally, I would like to drop resolution after some time. We e.g definitely do not need every trace after say two weeks but it would be nice to keep a few samples. The traces seem to be the ones using most storage. I know you can sample tracing in the agent. We would like to not have to do that...
A trace is actually a logical concept containing transaction and span documents, and potentially related error documents. All of these documents are stored in separated indices. Therefore if you would like to keep a sample of traces, one would need to figure out which root transactions (not having a parent_id) to keep and only delete unrelated documents from the other indices.
Unfortunately there is no out-of-the box logic at the moment taking care of this. As you already pointed out, the recommendation would be to enable sampling already in the agents.
Right, that is true. Thanks for pointing that out @simitt
I was toying with the idea of using some sort of conditional document routing based on maybe transaction.id to different indices. Something like all transaction.id s ending in a certain sequence, go to a special index, which will not be deleted by the ILM policy.
This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.