# Otel agent with apm server is giving error

**URL:** https://discuss.elastic.co/t/otel-agent-with-apm-server-is-giving-error/284616
**Category:** APM
**Tags:** java, open-telemetry
**Created:** [September 20, 2021, 11:33am UTC](https://discuss.elastic.co/t/otel-agent-with-apm-server-is-giving-error/284616 "2021-09-20T11:33:16Z")
**Posts on this page:** 6
**Page:** 1

<div class="post-metadata">

### Author: ![vamsikrishna\_medeti](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vamsikrishna_medeti/32/59108_2.png) [@vamsikrishna\_medeti](https://discuss.elastic.co/u/vamsikrishna_medeti)
#### Post date: [September 20, 2021, 11:33am UTC](https://discuss.elastic.co/t/otel-agent-with-apm-server-is-giving-error/284616/1 "2021-09-20T11:33:16Z")

</div>

Hi, I am trying to trace java application with open telemetry java agent 1.16.0 and with APM-server 7.14.1.  
I have set the below environment variables:

```auto
set OTEL_EXPORTER_OTLP_ENDPOINT=http://XX.XX.XX.XX:8200/
set OTEL_RESOURCE_ATTRIBUTES=service.name=Trail

```

The APM server is not secured so using http only. I am getting the below error in the java logs:

`[otel.javaagent 2021-09-20 16:54:55:585 +0530] [grpc-default-executor-1] WARN io.opentelemetry.exporter.otlp.trace.OtlpGrpcSpanExporter - Failed to expo rt spans. Error message: INTERNAL: http2 exception `  
Can you please help?

---

<div class="post-metadata">

### Author: ![Sylvain\_Juge](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sylvain_juge/32/55521_2.png) [@Sylvain\_Juge](https://discuss.elastic.co/u/Sylvain_Juge)
#### Post date: [September 20, 2021, 11:48am UTC](https://discuss.elastic.co/t/otel-agent-with-apm-server-is-giving-error/284616/2 "2021-09-20T11:48:14Z")

</div>

Hi @vamsikrishna_medeti ,

The issue seems definitely related to communication with backend server.

I'm not really familiar with debugging such issues with OTel agent, did you tried the debug logging from [https://github.com/open-telemetry/opentelemetry-java-instrumentation#troubleshooting](https://github.com/open-telemetry/opentelemetry-java-instrumentation#troubleshooting) ?  
Does it provides more information about the communication issue ?

---

<div class="post-metadata">

### Author: ![vamsikrishna\_medeti](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vamsikrishna_medeti/32/59108_2.png) [@vamsikrishna\_medeti](https://discuss.elastic.co/u/vamsikrishna_medeti)
#### Post date: [September 20, 2021, 12:07pm UTC](https://discuss.elastic.co/t/otel-agent-with-apm-server-is-giving-error/284616/3 "2021-09-20T12:07:22Z")

</div>

Its not showing any other exception or error in output even after enabling debug logs.  
But after setting the variable, `set OTEL_EXPERIMENTAL_EXPORTER_OTLP_PROTOCOL=http/protobuf`, I dont see that error.  
And I dont see any trace in the Kibana also. APM Server also not showing any logs. ☹

Here is the complete log:

```auto
D:\DevOps\github\metrics-exemplar-test>set OTEL_EXPERIMENTAL_EXPORTER_OTLP_PROTOCOL=http/protobuf
D:\DevOps\github\metrics-exemplar-test>set OTEL_TRACES_EXPORTER=otlp
D:\DevOps\github\metrics-exemplar-test>set OTEL_RESOURCE_ATTRIBUTES="service.name=Trail"
D:\DevOps\github\metrics-exemplar-test>set OTEL_EXPORTER_OTLP_ENDPOINT=http://xx.xx.xx.xxx:8200
D:\DevOps\github\metrics-exemplar-test>java -javaagent:C:\Users\fe265\Downloads\opentelemetry-javaagent-all-1.16.0.jar -jar target\rest-service-complete
-0.0.1-SNAPSHOT.jar
[otel.javaagent 2021-09-20 17:29:18:257 +0530] [main] INFO io.opentelemetry.javaagent.tooling.VersionLogger - opentelemetry-javaagent - version: 1.6.0

  . _______ _ _
 /\\ / ___'___ _ _(_)_ ____ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
 \\/ ___)| |_)| | | | | || (_| | ) ) ) )
  ' | ____| .__ |_| |_|_| |_\__, | / / / /
 =========|_|==============|___/=/_/_/_/
 :: Spring Boot :: (v2.5.2)

2021-09-20 17:29:23.065 INFO 14376 --- [main] c.e.restservice.RestServiceApplication : Starting RestServiceApplication v0.0.1-SNAPSHOT usi
ng Java 1.8.0_271 on NHHYDL-00217 with PID 14376 (D:\DevOps\github\metrics-exemplar-test\target\rest-service-complete-0.0.1-SNAPSHOT.jar started by fe26
5 in D:\DevOps\github\metrics-exemplar-test)
2021-09-20 17:29:23.073 INFO 14376 --- [main] c.e.restservice.RestServiceApplication : No active profile set, falling back to default prof
iles: default
2021-09-20 17:29:24.806 INFO 14376 --- [main] o.s.b.w.embedded.tomcat.TomcatWebServer : Tomcat initialized with port(s): 8080 (http)
2021-09-20 17:29:24.882 INFO 14376 --- [main] o.apache.catalina.core.StandardService : Starting service [Tomcat]
2021-09-20 17:29:24.882 INFO 14376 --- [main] org.apache.catalina.core.StandardEngine : Starting Servlet engine: [Apache Tomcat/9.0.48]
2021-09-20 17:29:24.999 INFO 14376 --- [main] o.a.c.c.C.[Tomcat].[localhost].[/] : Initializing Spring embedded WebApplicationContext
2021-09-20 17:29:24.999 INFO 14376 --- [main] w.s.c.ServletWebServerApplicationContext : Root WebApplicationContext: initialization complete
d in 1862 ms
2021-09-20 17:29:25.793 INFO 14376 --- [main] o.s.b.w.embedded.tomcat.TomcatWebServer : Tomcat started on port(s): 8080 (http) with context
 path ''
2021-09-20 17:29:25.892 INFO 14376 --- [main] o.s.b.w.embedded.tomcat.TomcatWebServer : Tomcat initialized with port(s): 9999 (http)
2021-09-20 17:29:25.893 INFO 14376 --- [main] o.apache.catalina.core.StandardService : Starting service [Tomcat]
2021-09-20 17:29:25.893 INFO 14376 --- [main] org.apache.catalina.core.StandardEngine : Starting Servlet engine: [Apache Tomcat/9.0.48]
2021-09-20 17:29:25.901 INFO 14376 --- [main] o.a.c.c.C.[Tomcat-1].[localhost].[/] : Initializing Spring embedded WebApplicationContext
2021-09-20 17:29:25.901 INFO 14376 --- [main] w.s.c.ServletWebServerApplicationContext : Root WebApplicationContext: initialization complete
d in 99 ms
2021-09-20 17:29:25.916 INFO 14376 --- [main] o.s.b.a.e.web.EndpointLinksResolver : Exposing 13 endpoint(s) beneath base path '/actuato
r'
2021-09-20 17:29:25.966 INFO 14376 --- [main] o.s.b.w.embedded.tomcat.TomcatWebServer : Tomcat started on port(s): 9999 (http) with context
 path ''
2021-09-20 17:29:25.982 INFO 14376 --- [main] c.e.restservice.RestServiceApplication : Started RestServiceApplication in 3.436 seconds (JV
M running for 7.933)
2021-09-20 17:29:36.861 INFO 14376 --- [nio-8080-exec-1] o.a.c.c.C.[Tomcat].[localhost].[/] : Initializing Spring DispatcherServlet 'dispatcherSe
rvlet'
2021-09-20 17:29:36.861 INFO 14376 --- [nio-8080-exec-1] o.s.web.servlet.DispatcherServlet : Initializing Servlet 'dispatcherServlet'
2021-09-20 17:29:36.862 INFO 14376 --- [nio-8080-exec-1] o.s.web.servlet.DispatcherServlet : Completed initialization in 1 ms
GreetingController.greeting: Sleeping for 10 ms

```

---

<div class="post-metadata">

### Author: ![axw](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/axw/32/28197_2.png) [@axw](https://discuss.elastic.co/u/axw)
#### Post date: [September 20, 2021, 12:29pm UTC](https://discuss.elastic.co/t/otel-agent-with-apm-server-is-giving-error/284616/4 "2021-09-20T12:29:16Z")

</div>

> [@vamsikrishna\_medeti](#):
>
> OTEL\_EXPERIMENTAL\_EXPORTER\_OTLP\_PROTOCOL=http/protobuf

APM Server does not currently support OTLP/HTTP.

> The APM server is not secured so using http only.

Perhaps you need to also set `OTEL_EXPORTER_OTLP_INSECURE=true`? I'm not sure if the http scheme implies this for OTLP/GRPC; the docs suggest it does not.

---

<div class="post-metadata">

### Author: ![vamsikrishna\_medeti](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vamsikrishna_medeti/32/59108_2.png) [@vamsikrishna\_medeti](https://discuss.elastic.co/u/vamsikrishna_medeti)
#### Post date: [September 20, 2021, 2:31pm UTC](https://discuss.elastic.co/t/otel-agent-with-apm-server-is-giving-error/284616/5 "2021-09-20T14:31:04Z")

</div>

Its working fine with below env variables:

```auto
set OTEL_EXPORTER_OTLP_TRACES_PROTOCOL=grpc
set OTEL_TRACES_EXPORTER=otlp
set OTEL_RESOURCE_ATTRIBUTES=service.name=AAA
set OTEL_EXPORTER_OTLP_INSECURE=true

```

APM Server Version: 7.14.1  
Otel agent: 1.16.0  
Elasticsearch and kibana version: 7.9.0

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [October 11, 2021, 10:31am UTC](https://discuss.elastic.co/t/otel-agent-with-apm-server-is-giving-error/284616/6 "2021-10-11T10:31:06Z")

</div>

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.
