I'm using the EDOT for Java (elastic-otel-javaagent-1.6.0.jar) which automatically extracts traceparentid from incoming HTTP requests for distributed tracing. This works as expected, however, when our external users also use OTEL they are injecting their own traceparentid. This is causing issues as they often group multiple requests into one trace.
I believe it is possible to create our own implementation of TextMapPropagator.java as a Custom propagator. But this seems moderately complicated and we would need our web layer to inject additional HTTP headers, for example X-Forwarded-For (XFF), to distinguish between internal and external requests.
I assume this is a common problem, therefore it feels like there should exist a more straightforward solution.
Does anyone know an easy way to make EDOT for java not extract traceparentid when the request is from an external source?
Note: If I understand correctly, EDOT is essentially a thin wrapper around the OTEL Java agent. If questions about this type of functionality are better suited for an OTEL forum, please let me know.
Yes EDOT is a thin wrapper and a few features that the intent would be to push upstream.
But you can still certainly ask questions here.
The "legacy" Java APM agent has this feature which is what you are looking for, but I do not see an equivalent in OTEL at this time. I asked internally as well.
This is something that we plan to work on it in the following weeks/months.
While there is no ETA, this is definitely something we plan to address soon, in particular, we aim to provide the “restart” strategy which systematically starts a new trace when the header is already present.
From your description it seems that you need to have a more advanced solution though:
your service is handling both internal and eternal requests, which might be detected through the usage of X-Forwarded-For header
you need to be able to distinguish between internal and external, which seems to fit the “restart_external” strategy.
If you have the ability to modify your reverse-proxy configuration, I think that implementing the following would work:
when setting the X-Forwarded-For for external calls, remove the `traceparent` header if present.
for the internal call, do not alter the request headers
Thank you for your response. You are correct, we have applications which are both used internally and externally.
So the first feature you aim to provide would be the restart strategy and not the restart_external strategy?
What makes this more challenging is that we deploy all of our applications on WildFly. We attach the EDOT Java agent to the Wildfly instance, and it seems we cannot have EDOT specific configuration for each deployment/artifact. Therefore setting the restart strategy for the whole Wildfly won't work for us sadly. We have the same issue that I have not been able to give each deployment on the Wildfly different service.name.
I will look into having our web layer remove the traceparent header.
So the first feature you aim to provide would be the restart strategy and not the restart_external strategy?
Yes, the assumption is that the most common case only the services “on the edge” need this and it’s quite static, in the sense that we do not need to filter on a per-request basis whether or not the request is an internal or an external one.
In your case, it seems that the same JVM has both internal and external services, can you elaborate a bit how do you distinguish from those ? For example, do you use a known list of URL prefixes or an HTTP header ? Also, are the external services calling the internal ones with the JVM “calling itself” on another request path ?
Also, do you use a reverse-proxy to set the `X-Forwarded-For` headers before Wildfly ? or are external requests directly sent to Wildfly and you have to deal with those on Wildfly ?
There is currently a limitation of OpenTelemetry regarding the deployment of multiple applications on the same JVM, as they are all considered to be part of the same `service.name` and can’t be configured separately.
Yes, we have multiple applications running on the same JVM (because of Wildfly), with both internal and external APIs. We have a reverse-proxy, but we don't set X-Forwarded-For for all requests, only for specific applications.
We don't necessarily distinguish between internal and external applications except that the internal application URL/ports are not exposed outside of our network. Though, some endpoints receive both internal and external requests.
As internal requests almost never goes trough the reverse-proxy we will make the reverse-proxy remove the traceparent HTTP header for all requests and see if that works for us.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.