Creating APM index in Elasticsearch based on event name in APM server 8.7.1

[mralians]

I upgraded APM from version 7.17 to 8.7.1. In the previous version, the APM index in Elasticsearch was based on this APM server configuration:

- index: "apm-%{[observer.version]}-%{[service.name]}-metric-%{+yyyy.MM.dd}"
  when.contains:
    processor.event: "metric"

The roll-over policy was based on this index name, but this configuration is no longer working in the newer version (8.7.1). Can anyone tell me how to create an APM index in Elasticsearch based on the APM client event name in APM server version 8.7.1?

[axw (Andrew Wilkins)]

@mralians in 8.x, APM Server writes documents to data streams with a specific format: <type>-<dataset>-<namespace>. At the moment, the only thing that can be customised is the namespace. Everything else is based on the data.

Custom application metrics are sent to service-specific data streams.

Can you tell us a little bit about your reasons for splitting indices by service.name? Is it so you can keep metrics for some services longer than others?

[mralians]

In the previous version, I had APM indices based on the processor.name, for example, apm-transaction-2022.03. Based on this index name pattern, I could define rollover policies, which allowed me to obtain different APM indices based on the processor.name with different rollover policies.

can I achieve this in the APM Server version 8.x?

[axw (Andrew Wilkins)]

This is not currently possible. It is in our backlog, and there have been some recent changes to make it possible. I can't say when it will happen, but we do intend to add support for namespacing data streams.