Multiple Retention Policies for High vs Low Frequency Transactions

Elastic Observability APM
1

Hi all :slight_smile:

I have a class of transactions which are very low frequency compared to most of the transactions I will be collecting data on via APM, and I'd like to keep this data longer than the high frequency data.

Is there a way to do this?

I am only just beginning to learn how to operationalize the data that APM is collecting, so maybe I am asking the wrong questions. Any advice welcome!

2

Hi @dvisz-inf :slight_smile:

I have a class of transactions which are very low frequency compared to most of the transactions I will be collecting data on via APM, and I'd like to keep this data longer than the high frequency data.

Is there a way to do this?

It depends: do you know at ingestion time? Do you know from the transaction names (or some other properties? labels?) which data are more frequent than others?

One approach would be to route the data into different indices, say "apm-frequent-transactions" and "apm-infrequent-transactions". This can be done by modifying output.elasticsearch.indices in apm-server.yml to route the data based on the fields and values contained within. See Configure the Elasticsearch output | APM User Guide [8.11] | Elastic.

With the data in different indices, you can then create different ILM (Index Lifecycle Management) policies for each category. See https://www.elastic.co/guide/en/apm/server/current/manual-ilm-setup.html.

3

Yes, I know at ingest time. Or at least, the guess I have at ingest time is good enough!

Your solution seems perfect. Thanks!

4

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.