Otel agent with apm server is giving error

Elastic Observability APM
,
1

Hi, I am trying to trace java application with open telemetry java agent 1.16.0 and with APM-server 7.14.1.
I have set the below environment variables:

set OTEL_EXPORTER_OTLP_ENDPOINT=http://XX.XX.XX.XX:8200/
set OTEL_RESOURCE_ATTRIBUTES=service.name=Trail

The APM server is not secured so using http only. I am getting the below error in the java logs:

[otel.javaagent 2021-09-20 16:54:55:585 +0530] [grpc-default-executor-1] WARN io.opentelemetry.exporter.otlp.trace.OtlpGrpcSpanExporter - Failed to expo rt spans. Error message: INTERNAL: http2 exception
Can you please help?

2

Hi @vamsikrishna_medeti ,

The issue seems definitely related to communication with backend server.

I'm not really familiar with debugging such issues with OTel agent, did you tried the debug logging from https://github.com/open-telemetry/opentelemetry-java-instrumentation#troubleshooting ?
Does it provides more information about the communication issue ?

3

Its not showing any other exception or error in output even after enabling debug logs.
But after setting the variable, set OTEL_EXPERIMENTAL_EXPORTER_OTLP_PROTOCOL=http/protobuf, I dont see that error.
And I dont see any trace in the Kibana also. APM Server also not showing any logs. :frowning:

Here is the complete log:

D:\DevOps\github\metrics-exemplar-test>set OTEL_EXPERIMENTAL_EXPORTER_OTLP_PROTOCOL=http/protobuf
D:\DevOps\github\metrics-exemplar-test>set OTEL_TRACES_EXPORTER=otlp
D:\DevOps\github\metrics-exemplar-test>set OTEL_RESOURCE_ATTRIBUTES="service.name=Trail"
D:\DevOps\github\metrics-exemplar-test>set OTEL_EXPORTER_OTLP_ENDPOINT=http://xx.xx.xx.xxx:8200
D:\DevOps\github\metrics-exemplar-test>java -javaagent:C:\Users\fe265\Downloads\opentelemetry-javaagent-all-1.16.0.jar -jar target\rest-service-complete
-0.0.1-SNAPSHOT.jar
[otel.javaagent 2021-09-20 17:29:18:257 +0530] [main] INFO io.opentelemetry.javaagent.tooling.VersionLogger - opentelemetry-javaagent - version: 1.6.0

  .   ____          _            __ _ _
 /\\ / ___'_ __ _ _(_)_ __  __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
 \\/  ___)| |_)| | | | | || (_| |  ) ) ) )
  '  |____| .__|_| |_|_| |_\__, | / / / /
 =========|_|==============|___/=/_/_/_/
 :: Spring Boot ::                (v2.5.2)

2021-09-20 17:29:23.065  INFO 14376 --- [           main] c.e.restservice.RestServiceApplication   : Starting RestServiceApplication v0.0.1-SNAPSHOT usi
ng Java 1.8.0_271 on NHHYDL-00217 with PID 14376 (D:\DevOps\github\metrics-exemplar-test\target\rest-service-complete-0.0.1-SNAPSHOT.jar started by fe26
5 in D:\DevOps\github\metrics-exemplar-test)
2021-09-20 17:29:23.073  INFO 14376 --- [           main] c.e.restservice.RestServiceApplication   : No active profile set, falling back to default prof
iles: default
2021-09-20 17:29:24.806  INFO 14376 --- [           main] o.s.b.w.embedded.tomcat.TomcatWebServer  : Tomcat initialized with port(s): 8080 (http)
2021-09-20 17:29:24.882  INFO 14376 --- [           main] o.apache.catalina.core.StandardService   : Starting service [Tomcat]
2021-09-20 17:29:24.882  INFO 14376 --- [           main] org.apache.catalina.core.StandardEngine  : Starting Servlet engine: [Apache Tomcat/9.0.48]
2021-09-20 17:29:24.999  INFO 14376 --- [           main] o.a.c.c.C.[Tomcat].[localhost].[/]       : Initializing Spring embedded WebApplicationContext
2021-09-20 17:29:24.999  INFO 14376 --- [           main] w.s.c.ServletWebServerApplicationContext : Root WebApplicationContext: initialization complete
d in 1862 ms
2021-09-20 17:29:25.793  INFO 14376 --- [           main] o.s.b.w.embedded.tomcat.TomcatWebServer  : Tomcat started on port(s): 8080 (http) with context
 path ''
2021-09-20 17:29:25.892  INFO 14376 --- [           main] o.s.b.w.embedded.tomcat.TomcatWebServer  : Tomcat initialized with port(s): 9999 (http)
2021-09-20 17:29:25.893  INFO 14376 --- [           main] o.apache.catalina.core.StandardService   : Starting service [Tomcat]
2021-09-20 17:29:25.893  INFO 14376 --- [           main] org.apache.catalina.core.StandardEngine  : Starting Servlet engine: [Apache Tomcat/9.0.48]
2021-09-20 17:29:25.901  INFO 14376 --- [           main] o.a.c.c.C.[Tomcat-1].[localhost].[/]     : Initializing Spring embedded WebApplicationContext
2021-09-20 17:29:25.901  INFO 14376 --- [           main] w.s.c.ServletWebServerApplicationContext : Root WebApplicationContext: initialization complete
d in 99 ms
2021-09-20 17:29:25.916  INFO 14376 --- [           main] o.s.b.a.e.web.EndpointLinksResolver      : Exposing 13 endpoint(s) beneath base path '/actuato
r'
2021-09-20 17:29:25.966  INFO 14376 --- [           main] o.s.b.w.embedded.tomcat.TomcatWebServer  : Tomcat started on port(s): 9999 (http) with context
 path ''
2021-09-20 17:29:25.982  INFO 14376 --- [           main] c.e.restservice.RestServiceApplication   : Started RestServiceApplication in 3.436 seconds (JV
M running for 7.933)
2021-09-20 17:29:36.861  INFO 14376 --- [nio-8080-exec-1] o.a.c.c.C.[Tomcat].[localhost].[/]       : Initializing Spring DispatcherServlet 'dispatcherSe
rvlet'
2021-09-20 17:29:36.861  INFO 14376 --- [nio-8080-exec-1] o.s.web.servlet.DispatcherServlet        : Initializing Servlet 'dispatcherServlet'
2021-09-20 17:29:36.862  INFO 14376 --- [nio-8080-exec-1] o.s.web.servlet.DispatcherServlet        : Completed initialization in 1 ms
GreetingController.greeting: Sleeping for 10 ms
4

APM Server does not currently support OTLP/HTTP.

The APM server is not secured so using http only.

Perhaps you need to also set OTEL_EXPORTER_OTLP_INSECURE=true? I'm not sure if the http scheme implies this for OTLP/GRPC; the docs suggest it does not.

5

Its working fine with below env variables:

set OTEL_EXPORTER_OTLP_TRACES_PROTOCOL=grpc
set OTEL_TRACES_EXPORTER=otlp
set OTEL_RESOURCE_ATTRIBUTES=service.name=AAA
set OTEL_EXPORTER_OTLP_INSECURE=true

APM Server Version: 7.14.1
Otel agent: 1.16.0
Elasticsearch and kibana version: 7.9.0

6

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.