# OpenTelemetry Agent cannot connect to Elastic APM in Fleet

**URL:** https://discuss.elastic.co/t/opentelemetry-agent-cannot-connect-to-elastic-apm-in-fleet/347781
**Category:** APM
**Tags:** open-telemetry, java, server, docker
**Created:** [November 22, 2023, 5:45pm UTC](https://discuss.elastic.co/t/opentelemetry-agent-cannot-connect-to-elastic-apm-in-fleet/347781 "2023-11-22T17:45:54Z")
**Posts on this page:** 7
**Page:** 1

<div class="post-metadata">

### Author: ![mmarinov](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mmarinov/32/127937_2.png) [@mmarinov](https://discuss.elastic.co/u/mmarinov)
#### Post date: [November 22, 2023, 5:45pm UTC](https://discuss.elastic.co/t/opentelemetry-agent-cannot-connect-to-elastic-apm-in-fleet/347781/1 "2023-11-22T17:45:54Z")

</div>

**Kibana version** : 8.11.1

**Elasticsearch version** : 8.11.1

**APM Server version** : 8.11.1

**APM Agent language and version** : opentelemetry-javaagent.jar 1.32.0

**Browser version** : not relevant

**Original install method (e.g. download page, yum, deb, from source, etc.) and version**: Elastic Cloud

**Fresh install or upgraded from other version?** fresh

**Is there anything special in your setup?** No, I'm using the default setup

**Description of the problem including expected versus actual behavior. Please include screenshots (if relevant)**:

I'm trying to instrument my application with the `opentelemetry-javaagent.jar` agent and send the traces and metrics directly to the APM Server (without OpenTelemetry Collector in the middle), but I'm always facing the following error in the logs:

```auto
[otel.javaagent 2023-11-22 18:52:12:134 +0200] [OkHttp https://xxxx.apm.eu-central-1.aws.cloud.es.io/...] WARN io.opentelemetry.exporter.internal.grpc.GrpcExporter - Failed to export metrics. Server responded with gRPC status code 2. 
Error message: stream was reset: PROTOCOL_ERROR
[otel.javaagent 2023-11-22 18:52:12:688 +0200] [OkHttp https://xxxx.apm.eu-central-1.aws.cloud.es.io/...] WARN io.opentelemetry.exporter.internal.grpc.GrpcExporter - Failed to export logs. Server responded with gRPC status code 2. Err
or message: stream was reset: PROTOCOL_ERROR

```

At first I was trying to run this setup with the Elastic Stack, running locally using Docker. Then I signed up for a free trial of Elastic Cloud to rule out any configuration errors I could have made with my Docker setup.  
Then to rule out problems with my application I checked out the `opentelemetry-java-examples`.  
The result is always the same - the `opentelemetry-javaagent` cannot connect to the APM Server with the same error.

**Steps to reproduce** :

1. Setup Elastic APM Server (either with Elastic Cloud or using Docker, following this guide: [Getting started with the Elastic Stack and Docker Compose: Part 2 | Elastic Blog](https://www.elastic.co/blog/getting-started-with-the-elastic-stack-and-docker-compose-part-2))
2. Checkout the OpenTelemetry `javaagent` example, from OpenTelemetry's official GitHub repo: [https://github.com/open-telemetry/opentelemetry-java-examples/tree/main/javaagent](https://github.com/open-telemetry/opentelemetry-java-examples/tree/main/javaagent)  
Alternatively use any Java application.
3. Configure the OpenTelemetry Java Agent, as described in `{elastic_url}/app/home#/tutorial/apm`, which includes the following environment variables:

```auto
OTEL_EXPORTER_OTLP_ENDPOINT: <apm_server_endpoint>
OTEL_EXPORTER_OTLP_HEADERS: "Authorization=Bearer <secret_token>"
OTEL_METRICS_EXPORTER: otlp
OTEL_LOGS_EXPORTER: otlp
OTEL_RESOURCE_ATTRIBUTES: service.name=<app-name>,service.version=<app-version>,deployment.environment=production

```

1. Download the latest `opentelemetry-javaagent.jar` from [https://github.com/open-telemetry/opentelemetry-java-instrumentation/releases/latest/download/opentelemetry-javaagent.jar](https://github.com/open-telemetry/opentelemetry-java-instrumentation/releases/latest/download/opentelemetry-javaagent.jar)
2. Run start the Java application with `-javaagent:path/to/opentelemetry-javaagent.jar`

**Notes** :

1. I've tested this with Java 11 and Java 17 on both Windows and MacOS
2. Setting explicitly `OTEL_EXPORTER_OTLP_PROTOCOL=grpc`, does not solve the issue
3. Setting `OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf` leads to the following error:

```auto
[otel.javaagent 2023-11-22 19:43:26:160 +0200] [OkHttp https://xxxx.apm.eu-central-1.aws.cloud.es.io/...] ERROR io.opentelemetry.exporter.internal.http.HttpExporter - Failed to export logs. The request could not be executed. Full erro
r message: stream was reset: PROTOCOL_ERROR
okhttp3.internal.http2.StreamResetException: stream was reset: PROTOCOL_ERROR
        at okhttp3.internal.http2.Http2Stream.takeHeaders(Http2Stream.kt:148)
        at okhttp3.internal.http2.Http2ExchangeCodec.readResponseHeaders(Http2ExchangeCodec.kt:97)
        at okhttp3.internal.connection.Exchange.readResponseHeaders(Exchange.kt:110)
        at okhttp3.internal.http.CallServerInterceptor.intercept(CallServerInterceptor.kt:93)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
        at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.kt:34)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
        at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.kt:95)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
        at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.kt:83)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
        at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.kt:76)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
        at okhttp3.internal.connection.RealCall.getResponseWithInterceptorChain$okhttp(RealCall.kt:201)
        at okhttp3.internal.connection.RealCall$AsyncCall.run(RealCall.kt:517)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:750)
        Suppressed: okhttp3.internal.http2.StreamResetException: stream was reset: PROTOCOL_ERROR
                ... 18 more
        Suppressed: okhttp3.internal.http2.StreamResetException: stream was reset: REFUSED_STREAM
                ... 18 more
        Suppressed: okhttp3.internal.http2.StreamResetException: stream was reset: PROTOCOL_ERROR
                ... 18 more

```

---

<div class="post-metadata">

### Author: ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)
#### Post date: [November 23, 2023, 5:03am UTC](https://discuss.elastic.co/t/opentelemetry-agent-cannot-connect-to-elastic-apm-in-fleet/347781/2 "2023-11-23T05:03:18Z")

</div>

Hi @mmarinov Welcome to the community

**EDIT** I just saw something... from yours

`https://xxxx.apm.eu-central-1.aws.cloud.es.io/...] ERROR io.opentelemet`

You probably need to put the `:443` as often there are default ports such as `8200` that are NOT the default ports for Elastic Cloud. Also note in the tutorial config page the URL has `:443`

Here is my working version...

I have Elastic Cloud with APM 8.11.1  
MacOS Sonoma  
Java 17

I am using the Sample App from [here](https://github.com/open-telemetry/opentelemetry-java-examples/tree/main/javaagent)

I ran

`../gradlew bootJar`

Here is my docker-compose (I took out the collector as it is not needed)

```auto
version: '3'
services:
  app:
    build: ./
    environment:
      OTEL_SERVICE_NAME: "agent-example-app"
      OTEL_EXPORTER_OTLP_ENDPOINT: "https://myelasticdeploymentyadada.apm.us-west1.gcp.cloud.es.io:443/"
      # Logs are disabled by default
      OTEL_LOGS_EXPORTER: "otlp"
      OTEL_METRICS_EXPORTER: "otlp"
      OTEL_EXPORTER_OTLP_HEADERS: "Authorization=Bearer sadfasdfagskmL"
    ports:
      - "8080:8080"

```

then I Run

`docker-compose up --build`

And access...

`curl http://localhost:8080/ping`

and I see the Transaction show up...

What am I missing? ... I See the Transactions  
I do not see errors in the log.

 ![Screenshot 2023-11-22 at 8.58.24 PM](https://us1.discourse-cdn.com/elastic/original/3X/0/1/0192591832830929b7299a58afa8ce22ce553243.png)  
 ![Screenshot 2023-11-22 at 8.58.35 PM](https://us1.discourse-cdn.com/elastic/original/3X/7/9/79ae32bdac5ade087ae41abb357e0a122b97b4d2.png)  
 ![Screenshot 2023-11-22 at 8.58.39 PM](https://us1.discourse-cdn.com/elastic/original/3X/8/2/828743f20925dec510e3e034f4b5debd2b8c6f00.png)  
 ![Screenshot 2023-11-22 at 9.01.05 PM](https://us1.discourse-cdn.com/elastic/original/3X/e/8/e8d9747191fd7cd030b18c63c231a9b8e5bc684a.png)

I am even getting the correlated Logs

 ![Screenshot 2023-11-22 at 9.11.01 PM](https://us1.discourse-cdn.com/elastic/original/3X/9/1/9142b2a4a8531393813417078a7c098a4bdad8c9.png)

And Metrics

 ![Screenshot 2023-11-22 at 9.13.22 PM](https://us1.discourse-cdn.com/elastic/original/3X/9/5/954ba73e13174400a5667ab312f5664782585f3f.png)

---

<div class="post-metadata">

### Author: ![mmarinov](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mmarinov/32/127937_2.png) [@mmarinov](https://discuss.elastic.co/u/mmarinov)
#### Post date: [November 23, 2023, 10:48am UTC](https://discuss.elastic.co/t/opentelemetry-agent-cannot-connect-to-elastic-apm-in-fleet/347781/3 "2023-11-23T10:48:49Z")

</div>

Hi, @stephenb! Thank you for the quick reply!

The APM Server address does end with :443 so it's not the issue.

I was able to narrow down the cause (or at least the exact circumstances) for the issue: **All the time I was trying to run the application using IntelliJ** , which probably shifts the focus of my initial report a bit. If I build the application and run it outside of the IDE with `java -jar ...` (like in the docker container from the example) it really does work.

Here is the updated setup:

1. When I try to run a Java application (not even a Spring Boot one) with IntelliJ, instrumented with the OpenTelemetry Agent:  
 ![Screenshot 2023-11-23 at 12.13.24](https://us1.discourse-cdn.com/elastic/original/3X/5/5/559b037e248a26fc4aa9225e8180a40ddd81d39f.png)  
Environment variables:

```auto
OTEL_EXPORTER_OTLP_ENDPOINT=https://xxxx.apm.eu-central-1.aws.cloud.es.io:443;OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer xxxx";OTEL_LOGS_EXPORTER=otlp;OTEL_METRICS_EXPORTER=otlp;OTEL_RESOURCE_ATTRIBUTES=service.name=TimeLoopOTEL,service.version=1.0,deployment.environment=dev

```

I always get the error from my initial post.

1. However if I run the same application, instrumented with the Elastic APM Agent:

VM Options:

```auto
-javaagent:elastic-apm-agent.jar
-Delastic.apm.service_name=TimeLoop
-Delastic.apm.secret_token=xxxx
-Delastic.apm.server_url=https://xxxx.apm.eu-central-1.aws.cloud.es.io:443
-Delastic.apm.environment=dev
-Delastic.apm.application_packages=org.example

```

Environment variables: **none**

Everything works fine and I can see the service metrics in Kibana.

**The application itself is as simple as possible:**

- Main.java

```java
package org.example;

import java.time.Instant;

public class Main {

  public static void main(String[] args) {
    System.out.printf("Press Ctrl+C to quit.");

    while (true) {
      System.out.println("The time is: " + Instant.now().toString());
      try {
        Thread.sleep(5000);
      } catch (InterruptedException e) {
        System.out.println("Sleep was interrupted.");
        throw new RuntimeException(e);
      }
    }
  }
}

```

- build.gradle

```auto
plugins {
    id 'java'
}

group = 'org.example'
version = '1.0-SNAPSHOT'

repositories {
    mavenCentral()
}

dependencies {
    testImplementation platform('org.junit:junit-bom:5.9.1')
    testImplementation 'org.junit.jupiter:junit-jupiter'
}

test {
    useJUnitPlatform()
}

```

I understand that probably IntelliJ's own agent, that it uses to start the application from the IDE could be the cause of the issue, however it's still strange that it only affects the OTEL agent and does not affect the Elastic APM one.

---

<div class="post-metadata">

### Author: ![mmarinov](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mmarinov/32/127937_2.png) [@mmarinov](https://discuss.elastic.co/u/mmarinov)
#### Post date: [November 23, 2023, 3:19pm UTC](https://discuss.elastic.co/t/opentelemetry-agent-cannot-connect-to-elastic-apm-in-fleet/347781/4 "2023-11-23T15:19:42Z")

</div>

One more update: If I add the OpenTelemetry Collector between the OTEL Agent and the APM Server it works.

Here is the summary:

- Running a compiled JAR with `java -jar ... -javaagent:...` :

- Running the application from IntelliJ:

Probably one of the major differences with the Collector in place is that the connection to the collector is insecure.

---

<div class="post-metadata">

### Author: ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)
#### Post date: [November 23, 2023, 3:39pm UTC](https://discuss.elastic.co/t/opentelemetry-agent-cannot-connect-to-elastic-apm-in-fleet/347781/5 "2023-11-23T15:39:03Z")

</div>

@mmarinov sorry I can't help with IntelliJ I use VS code these days.

I might go check with the OTEL community or intelliJ community see if anyone knows anything over there.

My one question is are you absolutely sure that you're using the correct and latest open telemetry jar and Java version? In the IntelliJ

---

<div class="post-metadata">

### Author: ![mmarinov](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mmarinov/32/127937_2.png) [@mmarinov](https://discuss.elastic.co/u/mmarinov)
#### Post date: [November 23, 2023, 6:23pm UTC](https://discuss.elastic.co/t/opentelemetry-agent-cannot-connect-to-elastic-apm-in-fleet/347781/6 "2023-11-23T18:23:24Z")

</div>

@stephenb The opentelemetry-javaagent and the Java SDK are the same in all test cases and are the latest versions (actually the Java SDK may be several minor versions behind - I will check with the latest version as well).  
Thanks for you time!  
I will write back if I find a solution.

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [December 21, 2023, 6:23pm UTC](https://discuss.elastic.co/t/opentelemetry-agent-cannot-connect-to-elastic-apm-in-fleet/347781/7 "2023-12-21T18:23:40Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
