Kibana version: 8.15.0
Elasticsearch version: 8.15.0
APM Server version: 8.15.0
APM Agent language and version: Elastic-otel-Java agent ; jar file , Release 1.0.0
Original install method (e.g. download page, yum, deb, from source, etc.) and version: elastic-otel-agentextension/1.0.0
Is there anything special in your setup? For example, are you using the Logstash or Kafka outputs? Are you using a load balancer in front of the APM Servers? Have you changed index pattern, generated custom templates, changed agent configuration etc. NO
Description of the problem including expected versus actual behavior. Please include screenshots (if relevant):
We don't seethe Environment, namespace, being set though it is given as an attribute in the properties.
JAVA_PROPERTIES+=" -Dotel.resource.attributes=deployment.environment=${ENVIRONMENT}"
JAVA_PROPERTIES+=" -Dotel.resource.attributes=kubernetes.namespace=${KUBE_POD_NAMESPACE}"
JAVA_PROPERTIES+=" -Dotel.resource.attributes=kubernetes.pod=${KUBE_POD_NAME}"
JAVA_PROPERTIES+=" -Dotel.exporter.otlp.protocol=http/protobuf"
JAVA_PROPERTIES+=" -Dotel.instrumentation.jetty.enabled=true"
JAVA_PROPERTIES+=" -Dotel.instrumentation.apache-httpclient.enabled=true"
JAVA_PROPERTIES+=" -Dotel.exporter.otlp.endpoint=${ELASTIC_OTEL_AGENT_SERVER_URL}"
JAVA_PROPERTIES+=" -Dotel.instrumentation.methods.include="\
Also is this the right way to include custom methods?
JAVA_PROPERTIES+=" -Dotel.instrumentation.methods.include="
1.See the attached stack trace for instrumentation,
And
2. Dont see any label in kibana set for the transaction for environment.
Hello good morning,
Wonder of anyone has an answer to this , otherwise, please let m know if this is a bug that needs to be worked on, so that I could submit it as bug on the elastic-otel-java git repo.
Hi,
In your configuration you have set multiple values for otel.resource.attributes, so the last one will win.
Here you need to set all the 3 resource attributes in the same value as a comma-separated list.
For now, the labels.code_stacktrace comes from the span-stacktrace module that is enabled in our agent, it does not relates to otel.instrumentation.methods.include.
For capturing custom methods with configuration, I don't know and I need to do a bit of research. I think it's possible but not 100% sure yet, however you can already add a few annotations using the OpenTelemetry API as an alternative if you can modify the application.
Thank you, I will comma-seperate all the attributes for "otel.resource.attributes" and let you know.
For now, the labels.code_stacktrace comes from the span-stacktrace module that is enabled in our agent, it does not relates to otel.instrumentation.methods.include .
So, any reason why i see that stacktrace?
however you can already add a few annotations using the OpenTelemetry API as an alternative if you can modify the application
As of now, we can't modify the application, I can find that out.
That's correct, and I am now able to set different labels and the service.environment. Thank you for the good suggestion.
Regarding
For capturing custom methods with configuration, I don't know and I need to do a bit of research.
I cannot modify the code, is there any way this property otel.instrumentation.methods.include. be added to the list of the Java properties, for elastic_otel? As you are already providing this in the elastic apm_agent java properties. That would be a great help.
When using the elastic distribution of opentelemetry (EDOT), you get the same implementation as the original opentelemetry agent plus few extras, so it should work exactly the same.
So here you should be able to use otel.instrumentation.methods.include to capture a custom method without modifying code, however the syntax is not the same as the one used with Elastic APM agent.
That helped, I am now able to instrument custom methods.
COuld oyu please help me with understanding what this stack trace is about and how to get rid of it?
labels.code_stacktrace
at io.opentelemetry.javaagent.shaded.instrumentation.api.instrumenter.Instrumenter.doEnd(Instrumenter.java:263) at io.opentelemetry.javaagent.shaded.instrumentation.api.instrumenter.Instrumenter.end(Instrumenter.java:150) at io.opentelemetry.javaagent.instrumentation.servlet.ServletHelper.end(ServletHelper.java:62) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:124) at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:548) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:600) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235) at org.eclipse.jetty.server.session
Hi,
As written in feature documentation here you should set otel.java.experimental.span-stacktrace.min.duration=-1 to disable this feature.
It is not a "real" stacktrace in the sense that it does not comes from an application exception, it's just a way for the agent to capture the calling context of the span.
