Use APM with other indexes

[digitalron (Ronald Tumulak)]

Hi all,

I tried looking for references but couldn't find any, any help is appreciated...

Is there a way to point the APM page/view to other similar indexes? The use case is that we'd like to copy specific transactions and spans to a separate index for use in a later analysis, either manually or automatically as a result of ingest processing.

Currently, it seems the APM view can only read from apm-* but we'd like to copy selected transactions to something like analysis_apm so that this can be a more long-term index as opposed to apm-* which we'd apply some ILM on.

Many thanks,

Ronald

[gil (Gil Raphaelli)]

Hello again @digitalron. Two thoughts for you:

1. Configure Kibana to include those indices, multiple indices can be comma separated eg apm_oss.transactionIndices: "apm-*,archive-apm-*"
2. Alias any long term indices to match the pattern used by the UI

[felixbarny (Felix Barnsteiner)]

@digitalron

The use case is that we'd like to copy specific transactions and spans to a separate index for use in a later analysis

Seems like you want to store "interesting" traces and delete others. Is that right? We are also thinking about how we can better support use cases like that. Howe would your logic look like which traces you want to store for a longer period?

[digitalron (Ronald Tumulak)]

Hi Gil, thanks for the hints. I'm on commute so I'll try these tomorrow when I get to work.

Hi Felix, that's correct, we'd like to be able to store the "interesting" traces.
As for workflow, we see two ways:

1. An additional "Actions" entry like "Save" so that we can click on it when we see the trace we'd like to save. It'd be nice to have description and tags fields available for annotating.
2. For the ingest process, we could send a copy of the trace to an arbitrary predetermined, pre-setup index. The trace selection is something we can determine using some pipeline logic (an example is to base it on tags populated by the instrumented app, or thresholds for a particular span/transaction like duration time, etc)

Then, on the APM view, we can select the index we'd like to view, whether it's the "Save Trace" index or the index configured in the ingest (similar to how we can select the index in the Discover or Visualize pages).

Thanks!