APM transaction name grouping

Elastic Observability APM
1

Kibana version: 8.17.6

Elasticsearch version: 8.17.6

APM Server version: 8.17.6

APM Agent language and version: Elastic OTEL Java agent 1.4.1

Browser version: Chrome Version 137.0.7151.120 (Official Build) (64-bit)

Original install method (e.g. download page, yum, deb, from source, etc.) and version: deb

Fresh install or upgraded from other version? Fresh

Is there anything special in your setup?

We use the system property "ELASTIC.APM.USE_PATH_AS_TRANSACTION_NAME" set to true for the Java agent

Description of the problem including expected versus actual behavior. Please include screenshots (if relevant):

Our application performs HTTP POST calls to /ws/xlocate and /ws/xroute. However, in APM, in the Transactions tab of this particular service. These calls are grouped together in POST /ws/*. We would like to have the more detailed URL in the overview. The system property "ELASTIC.APM.USE_PATH_AS_TRANSACTION_NAME" is already set.

How can we archieve this ?

Steps to reproduce:

  1. In APM, click "Service Inventory"
  2. Click the Transactions tab
  3. Look at the Transactions list

Errors in browser console (if relevant): No

Provide logs and/or server output (if relevant):

2

Hi !

You are using the EDOT Java agent, and the ELASTIC_APM_USE_PATH_AS_TRANSACTION_NAME config option is only supported for the "APM Java agent".

The span name comes from the OpenTelemetry instrumentation and is designed to prevent high-cardinality, so if your request is mapped to /ws/{param} and you only have two distinct value of {param} then the instrumentation can't guess and have to assume that a potentially infinite number of values could be used.

When inspecting the documents captured for the transaction, do you see the complete path in the http.* attributes ? If yes, then it may be possible to use an ingest pipeline (or another transformation in the ingestion pipeline) to transform the span name.

Also, if you can share the json document of the transaction/span, it would help to know the instrumentation that captured it, maybe there is a config option that could allow to change this behavior.

EDIT: the http.route otel semconv attribute must be captured by the agent, so it could be used to rename the span (it should be mapped to either http.route or labels.http_route depending on how you ingest data).

3 
{
  "resourceSpans" : [ {
    "resource" : {
      "attributes" : [ {
        "key" : "container.id",
        "value" : {
          "stringValue" : "f787235e3fe7fa5933fd23c10af6c0c29bbf62a9d55f2fbb7bf23e9c7596ab99"
        }
      }, {
        "key" : "data_stream.namespace",
        "value" : {
          "stringValue" : "TS1"
        }
      }, {
        "key" : "deployment.environment",
        "value" : {
          "stringValue" : "TS1"
        }
      }, {
        "key" : "deployment.environment.name",
        "value" : {
          "stringValue" : "TS1"
        }
      }, {
        "key" : "host.arch",
        "value" : {
          "stringValue" : "amd64"
        }
      }, {
        "key" : "host.name",
        "value" : {
          "stringValue" : "geolocation-routing-6d85ff9b76-5mxxb"
        }
      }, {
        "key" : "os.description",
        "value" : {
          "stringValue" : "Linux 5.4.0-81-generic"
        }
      }, {
        "key" : "os.type",
        "value" : {
          "stringValue" : "linux"
        }
      }, {
        "key" : "process.command_args",
        "value" : {
          "arrayValue" : {
            "values" : [ {
              "stringValue" : "/opt/java/openjdk/bin/java"
            }, {
              "stringValue" : "-Xms256m"
            }, {
              "stringValue" : "-Xmx512m"
            }, {
              "stringValue" : "-XX:MaxMetaspaceSize=128m"
            }, {
              "stringValue" : "-Dfile.encoding=UTF-8"
            }, {
              "stringValue" : "-Duser.timezone=UTC"
            }, {
              "stringValue" : "-javaagent:/dd-agent.jar"
            }, {
              "stringValue" : "-javaagent:/elastic-otel-javaagent.jar"
            }, {
              "stringValue" : "-Ddd.version=latest"
            }, {
              "stringValue" : "-Ddd.trace.sample.rate=1"
            }, {
              "stringValue" : "-cp"
            }, {
              "stringValue" : "/app/resources:/app/classes:/app/libs/geolocation-core-0.0.1-SNAPSHOT.jar:/app/libs/spring-boot-starter-validation-3.5.0.jar:/app/libs/tomcat-embed-el-10.1.41.jar:/app/libs/hibernate-validator-8.0.2.Final.jar:/app/libs/jakarta.validation-api-3.0.2.jar:/app/libs/jboss-logging-3.6.1.Final.jar:/app/libs/classmate-1.7.0.jar:/app/libs/commons-lang3-3.17.0.jar:/app/libs/geolocation-legacy-api-0.0.1-SNAPSHOT.jar:/app/libs/spring-ws-core-4.1.0.jar:/app/libs/jakarta.xml.soap-api-3.0.2.jar:/app/libs/spring-aop-6.2.7.jar:/app/libs/spring-beans-6.2.7.jar:/app/libs/spring-oxm-6.2.7.jar:/app/libs/spring-web-6.2.7.jar:/app/libs/spring-webmvc-6.2.7.jar:/app/libs/jaxb-runtime-4.0.5.jar:/app/libs/jaxb-core-4.0.5.jar:/app/libs/txw2-4.0.5.jar:/app/libs/istack-commons-runtime-4.1.2.jar:/app/libs/spring-boot-starter-web-3.5.0.jar:/app/libs/spring-boot-starter-json-3.5.0.jar:/app/libs/jackson-datatype-jdk8-2.19.0.jar:/app/libs/jackson-module-parameter-names-2.19.0.jar:/app/libs/spring-boot-starter-undertow-3.5.0.jar:/app/libs/undertow-core-2.3.18.Final.jar:/app/libs/xnio-api-3.8.16.Final.jar:/app/libs/wildfly-common-1.5.4.Final.jar:/app/libs/wildfly-client-config-1.0.1.Final.jar:/app/libs/xnio-nio-3.8.16.Final.jar:/app/libs/jboss-threads-3.5.0.Final.jar:/app/libs/undertow-servlet-2.3.18.Final.jar:/app/libs/jakarta.servlet-api-6.0.0.jar:/app/libs/undertow-websockets-jsr-2.3.18.Final.jar:/app/libs/jakarta.websocket-api-2.1.1.jar:/app/libs/jakarta.websocket-client-api-2.1.1.jar:/app/libs/geolocation-ptv-provider-0.0.1-SNAPSHOT.jar:/app/libs/geolocation-api-0.0.1-SNAPSHOT.jar:/app/libs/spring-boot-starter-actuator-3.5.0.jar:/app/libs/spring-boot-starter-3.5.0.jar:/app/libs/spring-boot-3.5.0.jar:/app/libs/spring-boot-autoconfigure-3.5.0.jar:/app/libs/spring-boot-starter-logging-3.5.0.jar:/app/libs/logback-classic-1.5.18.jar:/app/libs/logback-core-1.5.18.jar:/app/libs/log4j-to-slf4j-2.24.3.jar:/app/libs/log4j-api-2.24.3.jar:/app/libs/jul-to-slf4j-2.0.17.jar:/app/libs/jakarta.annotation-api-2.1.1.jar:/app/libs/snakeyaml-2.4.jar:/app/libs/spring-boot-actuator-autoconfigure-3.5.0.jar:/app/libs/spring-boot-actuator-3.5.0.jar:/app/libs/jackson-datatype-jsr310-2.19.0.jar:/app/libs/micrometer-observation-1.15.0.jar:/app/libs/micrometer-commons-1.15.0.jar:/app/libs/micrometer-jakarta9-1.15.0.jar:/app/libs/micrometer-core-1.15.0.jar:/app/libs/HdrHistogram-2.2.2.jar:/app/libs/LatencyUtils-2.0.3.jar:/app/libs/spring-xml-4.1.0.jar:/app/libs/saaj-impl-3.0.4.jar:/app/libs/stax-ex-2.1.0.jar:/app/libs/jakarta.activation-api-2.1.3.jar:/app/libs/spring-context-6.2.7.jar:/app/libs/spring-expression-6.2.7.jar:/app/libs/spring-boot-configuration-processor-3.5.0.jar:/app/libs/spring-cloud-starter-4.3.0.jar:/app/libs/spring-cloud-context-4.3.0.jar:/app/libs/spring-security-crypto-6.5.0.jar:/app/libs/spring-cloud-commons-4.3.0.jar:/app/libs/bcprov-jdk18on-1.80.jar:/app/libs/logstash-logback-encoder-8.1.jar:/app/libs/jackson-databind-2.19.0.jar:/app/libs/jackson-annotations-2.19.0.jar:/app/libs/jackson-core-2.19.0.jar:/app/libs/slf4j-api-2.0.17.jar:/app/libs/jakarta.xml.bind-api-4.0.2.jar:/app/libs/spring-core-6.2.7.jar:/app/libs/spring-jcl-6.2.7.jar"
            }, {
              "stringValue" : "com.company.geolocationrouting.app.Application"
            } ]
          }
        }
      }, {
        "key" : "process.executable.path",
        "value" : {
          "stringValue" : "/opt/java/openjdk/bin/java"
        }
      }, {
        "key" : "process.pid",
        "value" : {
          "intValue" : "1"
        }
      }, {
        "key" : "process.runtime.description",
        "value" : {
          "stringValue" : "Eclipse Adoptium OpenJDK 64-Bit Server VM 21.0.1+12-LTS"
        }
      }, {
        "key" : "process.runtime.name",
        "value" : {
          "stringValue" : "OpenJDK Runtime Environment"
        }
      }, {
        "key" : "process.runtime.version",
        "value" : {
          "stringValue" : "21.0.1+12-LTS"
        }
      }, {
        "key" : "service.instance.id",
        "value" : {
          "stringValue" : "0130ba2e-7dad-42e4-a7e7-785d883edc4c"
        }
      }, {
        "key" : "service.name",
        "value" : {
          "stringValue" : "geolocation-routing"
        }
      }, {
        "key" : "telemetry.distro.name",
        "value" : {
          "stringValue" : "elastic"
        }
      }, {
        "key" : "telemetry.distro.version",
        "value" : {
          "stringValue" : "1.4.1"
        }
      }, {
        "key" : "telemetry.sdk.language",
        "value" : {
          "stringValue" : "java"
        }
      }, {
        "key" : "telemetry.sdk.name",
        "value" : {
          "stringValue" : "opentelemetry"
        }
      }, {
        "key" : "telemetry.sdk.version",
        "value" : {
          "stringValue" : "1.49.0"
        }
      } ]
    },
    "scopeSpans" : [ {
      "scope" : {
        "name" : "io.opentelemetry.java-http-client",
        "version" : "2.15.0-alpha"
      },
      "spans" : [ {
        "traceId" : "838b70ef5066816fe818970a72a63542",
        "spanId" : "ff6d5f31c7dc2e12",
        "parentSpanId" : "",
        "flags" : 257,
        "name" : "POST /ws/*",
        "kind" : 2,
        "startTimeUnixNano" : "1750300099604162348",
        "endTimeUnixNano" : "1750300099643153718",
        "attributes" : [ {
          "key" : "http.route",
          "value" : {
            "stringValue" : "/ws/*"
          }
        }, {
          "key" : "http.request.method",
          "value" : {
            "stringValue" : "POST"
          }
        }, {
          "key" : "code.stacktrace",
          "value" : {
            "stringValue" : "\tat io.opentelemetry.javaagent.shaded.instrumentation.api.instrumenter.Instrumenter.doEnd(Instrumenter.java:273)\n\tat io.opentelemetry.javaagent.shaded.instrumentation.api.instrumenter.Instrumenter.end(Instrumenter.java:151)\n\tat io.opentelemetry.javaagent.instrumentation.undertow.UndertowHelper.end(UndertowHelper.java:42)\n\tat io.opentelemetry.javaagent.instrumentation.undertow.UndertowHelper.exchangeCompleted(UndertowHelper.java:66)\n\tat io.opentelemetry.javaagent.instrumentation.undertow.EndSpanListener.exchangeEvent(EndSpanListener.java:23)\n\tat io.undertow.server.HttpServerExchange$ExchangeCompleteNextListener.proceed(HttpServerExchange.java:1982)\n\tat io.undertow.server.handlers.GracefulShutdownHandler$GracefulShutdownListener.exchangeEvent(GracefulShutdownHandler.java:196)\n\tat io.undertow.server.HttpServerExchange.invokeExchangeCompleteListeners(HttpServerExchange.java:1359)\n\tat io.undertow.server.HttpServerExchange.terminateResponse(HttpServerExchange.java:1646)\n\tat io.undertow.server.Connectors.terminateResponse(Connectors.java:184)\n\tat io.undertow.server.protocol.http.ServerFixedLengthStreamSinkConduit.channelFinished(ServerFixedLengthStreamSinkConduit.java:58)\n\tat io.undertow.conduits.AbstractFixedLengthStreamSinkConduit.exitFlush(AbstractFixedLengthStreamSinkConduit.java:316)\n\tat io.undertow.conduits.AbstractFixedLengthStreamSinkConduit.flush(AbstractFixedLengthStreamSinkConduit.java:234)\n\tat org.xnio.conduits.ConduitStreamSinkChannel.flush(ConduitStreamSinkChannel.java:162)\n\tat io.undertow.channels.DetachableStreamSinkChannel.flush(DetachableStreamSinkChannel.java:119)\n\tat org.xnio.channels.Channels.flushBlocking(Channels.java:63)\n\tat io.undertow.servlet.spec.ServletOutputStreamImpl.flushInternal(ServletOutputStreamImpl.java:526)\n\tat io.undertow.servlet.spec.ServletOutputStreamImpl.flush(ServletOutputStreamImpl.java:500)\n\tat org.springframework.ws.transport.TransportOutputStream.flush(TransportOutputStream.java:58)\n\tat org.springframework.ws.soap.saaj.SaajSoapMessage.writeTo(SaajSoapMessage.java:269)\n\tat org.springframework.ws.transport.AbstractWebServiceConnection.send(AbstractWebServiceConnection.java:46)\n\tat org.springframework.ws.transport.support.WebServiceMessageReceiverObjectSupport.handleConnection(WebServiceMessageReceiverObjectSupport.java:99)\n\tat org.springframework.ws.transport.http.WebServiceMessageReceiverHandlerAdapter.handle(WebServiceMessageReceiverHandlerAdapter.java:63)\n\tat org.springframework.ws.transport.http.MessageDispatcherServlet.doService(MessageDispatcherServlet.java:314)\n\tat org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1014)\n\tat org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:914)\n\tat jakarta.servlet.http.HttpServlet.service(HttpServlet.java:547)\n\tat org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:885)\n\tat jakarta.servlet.http.HttpServlet.service(HttpServlet.java:614)\n\tat io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:74)\n\tat io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:129)\n\tat com.company.geolocationrouting.api.legacy.WsdlQuestionMarkSuffixFilter.doFilter(WsdlQuestionMarkSuffixFilter.java:50)\n\tat io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:67)\n\tat io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)\n\tat org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100)\n\tat org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)\n\tat io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:67)\n\tat io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)\n\tat org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93)\n\tat org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)\n\tat io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:67)\n\tat io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)\n\tat org.springframework.web.servlet.v6_0.OpenTelemetryHandlerMappingFilter.doFilter(OpenTelemetryHandlerMappingFilter.java:78)\n\tat io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:67)\n\tat io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)\n\tat org.springframework.web.filter.ServerHttpObservationFilter.doFilterInternal(ServerHttpObservationFilter.java:114)\n\tat org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)\n\tat io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:67)\n\tat io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)\n\tat org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201)\n\tat org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)\n\tat io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:67)\n\tat io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)\n\tat io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:84)\n\tat io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62)\n\tat io.undertow.servlet.handlers.ServletChain$1.handleRequest(ServletChain.java:68)\n\tat io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36)\n\tat io.undertow.servlet.handlers.RedirectDirHandler.handleRequest(RedirectDirHandler.java:68)\n\tat io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:117)\n\tat io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57)\n\tat io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)\n\tat io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46)\n\tat io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64)\n\tat io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60)\n\tat io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77)\n\tat io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43)\n\tat io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)\n\tat io.undertow.servlet.handlers.SendErrorPageHandler.handleRequest(SendErrorPageHandler.java:52)\n\tat io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)\n\tat io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:276)\n\tat io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135)\n\tat io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:132)\n\tat io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48)\n\tat io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)\n\tat io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:256)\n\tat io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:101)\n\tat io.undertow.server.Connectors.executeRootHandler(Connectors.java:395)\n\tat io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:861)\n\tat org.jboss.threads.ContextHandler$1.runWith(ContextHandler.java:18)\n\tat org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2513)\n\tat org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1538)\n\tat org.xnio.XnioWorker$WorkerThreadFactory$1$1.run(XnioWorker.java:1282)\n\tat java.base/java.lang.Thread.run(Thread.java:1583)\n"
          }
        }, {
          "key" : "url.query",
          "value" : {
            "stringValue" : "wsdl"
          }
        }, {
          "key" : "url.path",
          "value" : {
            "stringValue" : "/ws/xlocate"
          }
        }, {
          "key" : "server.address",
          "value" : {
            "stringValue" : "geolocation-routing-test.company.com"
          }
        }, {
          "key" : "client.address",
          "value" : {
            "stringValue" : "xxx.xxx.xxx.xxx"
          }
        }, {
          "key" : "network.peer.address",
          "value" : {
            "stringValue" : "xxx.xxx.xxx.xxx"
          }
        }, {
          "key" : "http.response.status_code",
          "value" : {
            "intValue" : "200"
          }
        }, {
          "key" : "thread.id",
          "value" : {
            "intValue" : "44"
          }
        }, {
          "key" : "network.protocol.version",
          "value" : {
            "stringValue" : "1.1"
          }
        }, {
          "key" : "user_agent.original",
          "value" : {
            "stringValue" : "http-api/1.39"
          }
        }, {
          "key" : "url.scheme",
          "value" : {
            "stringValue" : "https"
          }
        }, {
          "key" : "thread.name",
          "value" : {
            "stringValue" : "XNIO-1 I/O-1"
          }
        }, {
          "key" : "network.peer.port",
          "value" : {
            "intValue" : "xxxxx"
          }
        } ],
        "status" : { }
      }, 	
    "schemaUrl" : "https://opentelemetry.io/schemas/1.24.0"
  } ]
}
4

I use the following ingestion pipeline:

[
  {
    "set": {
      "field": "transaction.name",
      "value": "{{{http.request.method}}} {{{url.path}}}",
      "if": "ctx.http?.request?.method != null && (ctx.http.request.method == 'POST' || ctx.http.request.method == 'GET') && ctx.url?.path != null && ctx.url.path != ''\r "
    }
  },
  {
    "set": {
      "field": "span.name",
      "value": "{{{http.request.method}}} {{{url.path}}}",
      "if": "ctx.http?.request?.method != null && (ctx.http.request.method == 'POST' || ctx.http.request.method == 'GET') && ctx.url?.path != null && ctx.url.path != ''\r "
    }
  },
  {
    "set": {
      "field": "labels.http_route",
      "value": "{{{url.path}}}",
      "if": "ctx.transaction?.type != null && ctx.transaction?.type == 'request'\r "
    }
  }
]

However, still, in the Transaction overview of a service, the "grouped" name is used, for example: "/ws/*" instead of /ws/xlocate which is in the url.path field.

Any ideas ?

5

The downside of ingestion pipeline is that you have to apply it to every type of document that potentially uses the span.name (or transaction.name), for example you will have to handle this for every type of document (traces, metrics, logs, ...) where this field is being used.

A simpler approach would be to modify the produced data earlier in the whole pipeline:

  • at the EDOT Java level by using an extension which allows to post-process the data directly into the agent, you will have to build maintain and deploy a simple extension for it.
  • use an OTTL transformation with an intermediate collector instance.

I am not familiar with OTTL at all, but I can help you with the agent extension option if needed.

6

As I understand, the Ingestion pipeline is executed before the document is indexed. However, as stated above, I modify the span.name and transaction.name to include the full path and even then, the UI is displaying it as "/ws/*".
I did a search over every index in Elastic and the "/ws/*" String is nowhere.
Does the UI "calculate" this value when the page is rendered ?

7

You are correct, the ingestion pipeline is executed before the document is indexed, however , the data sent to the otel collector is processed and produces more than one ES document, so each of them have to be modified.

This requires to know a bit more about the APM data model and when there are aggregations it might not be always possible to do that, for example: if data is aggregated on the span.name attribute and the other attributes are not stored with the aggregate document, then you won't be able to split this aggregate as only a single value (the span name you want to replace) will be present.

In short, it's probably simpler and more efficient to modify the data through OTTL or at the EDOT level when possible.