OpenTelemetry Agent cannot connect to Elastic APM in Fleet

[mmarinov (Miroslav Marinov)]

Kibana version: 8.11.1

Elasticsearch version: 8.11.1

APM Server version: 8.11.1

APM Agent language and version: opentelemetry-javaagent.jar 1.32.0

Browser version: not relevant

Original install method (e.g. download page, yum, deb, from source, etc.) and version: Elastic Cloud

Fresh install or upgraded from other version? fresh

Is there anything special in your setup? No, I'm using the default setup

Description of the problem including expected versus actual behavior. Please include screenshots (if relevant):

I'm trying to instrument my application with the opentelemetry-javaagent.jar agent and send the traces and metrics directly to the APM Server (without OpenTelemetry Collector in the middle), but I'm always facing the following error in the logs:

[otel.javaagent 2023-11-22 18:52:12:134 +0200] [OkHttp https://xxxx.apm.eu-central-1.aws.cloud.es.io/...] WARN io.opentelemetry.exporter.internal.grpc.GrpcExporter - Failed to export metrics. Server responded with gRPC status code 2. 
Error message: stream was reset: PROTOCOL_ERROR
[otel.javaagent 2023-11-22 18:52:12:688 +0200] [OkHttp https://xxxx.apm.eu-central-1.aws.cloud.es.io/...] WARN io.opentelemetry.exporter.internal.grpc.GrpcExporter - Failed to export logs. Server responded with gRPC status code 2. Err
or message: stream was reset: PROTOCOL_ERROR

At first I was trying to run this setup with the Elastic Stack, running locally using Docker. Then I signed up for a free trial of Elastic Cloud to rule out any configuration errors I could have made with my Docker setup.
Then to rule out problems with my application I checked out the opentelemetry-java-examples.
The result is always the same - the opentelemetry-javaagent cannot connect to the APM Server with the same error.

Steps to reproduce:

1. Setup Elastic APM Server (either with Elastic Cloud or using Docker, following this guide: Getting started with the Elastic Stack and Docker Compose: Part 2 | Elastic Blog)
2. Checkout the OpenTelemetry javaagent example, from OpenTelemetry's official GitHub repo: https://github.com/open-telemetry/opentelemetry-java-examples/tree/main/javaagent
   Alternatively use any Java application.
3. Configure the OpenTelemetry Java Agent, as described in {elastic_url}/app/home#/tutorial/apm, which includes the following environment variables:

OTEL_EXPORTER_OTLP_ENDPOINT: <apm_server_endpoint>
OTEL_EXPORTER_OTLP_HEADERS: "Authorization=Bearer <secret_token>"
OTEL_METRICS_EXPORTER: otlp
OTEL_LOGS_EXPORTER: otlp
OTEL_RESOURCE_ATTRIBUTES: service.name=<app-name>,service.version=<app-version>,deployment.environment=production

4. Download the latest opentelemetry-javaagent.jar from https://github.com/open-telemetry/opentelemetry-java-instrumentation/releases/latest/download/opentelemetry-javaagent.jar
5. Run start the Java application with -javaagent:path/to/opentelemetry-javaagent.jar

Notes:

1. I've tested this with Java 11 and Java 17 on both Windows and MacOS
2. Setting explicitly OTEL_EXPORTER_OTLP_PROTOCOL=grpc, does not solve the issue
3. Setting OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf leads to the following error:

[otel.javaagent 2023-11-22 19:43:26:160 +0200] [OkHttp https://xxxx.apm.eu-central-1.aws.cloud.es.io/...] ERROR io.opentelemetry.exporter.internal.http.HttpExporter - Failed to export logs. The request could not be executed. Full erro
r message: stream was reset: PROTOCOL_ERROR
okhttp3.internal.http2.StreamResetException: stream was reset: PROTOCOL_ERROR
        at okhttp3.internal.http2.Http2Stream.takeHeaders(Http2Stream.kt:148)
        at okhttp3.internal.http2.Http2ExchangeCodec.readResponseHeaders(Http2ExchangeCodec.kt:97)
        at okhttp3.internal.connection.Exchange.readResponseHeaders(Exchange.kt:110)
        at okhttp3.internal.http.CallServerInterceptor.intercept(CallServerInterceptor.kt:93)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
        at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.kt:34)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
        at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.kt:95)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
        at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.kt:83)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
        at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.kt:76)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
        at okhttp3.internal.connection.RealCall.getResponseWithInterceptorChain$okhttp(RealCall.kt:201)
        at okhttp3.internal.connection.RealCall$AsyncCall.run(RealCall.kt:517)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:750)
        Suppressed: okhttp3.internal.http2.StreamResetException: stream was reset: PROTOCOL_ERROR
                ... 18 more
        Suppressed: okhttp3.internal.http2.StreamResetException: stream was reset: REFUSED_STREAM
                ... 18 more
        Suppressed: okhttp3.internal.http2.StreamResetException: stream was reset: PROTOCOL_ERROR
                ... 18 more

[stephenb (Stephen Brown)]

Hi @mmarinov Welcome to the community

EDIT I just saw something... from yours

https://xxxx.apm.eu-central-1.aws.cloud.es.io/...] ERROR io.opentelemet

You probably need to put the :443 as often there are default ports such as 8200 that are NOT the default ports for Elastic Cloud. Also note in the tutorial config page the URL has :443

Here is my working version...

I have Elastic Cloud with APM 8.11.1
MacOS Sonoma
Java 17

I am using the Sample App from here

I ran

../gradlew bootJar

Here is my docker-compose (I took out the collector as it is not needed)

version: '3'
services:
  app:
    build: ./
    environment:
      OTEL_SERVICE_NAME: "agent-example-app"
      OTEL_EXPORTER_OTLP_ENDPOINT: "https://myelasticdeploymentyadada.apm.us-west1.gcp.cloud.es.io:443/"
      # Logs are disabled by default
      OTEL_LOGS_EXPORTER: "otlp"
      OTEL_METRICS_EXPORTER: "otlp"
      OTEL_EXPORTER_OTLP_HEADERS: "Authorization=Bearer sadfasdfagskmL"
    ports:
      - "8080:8080"

then I Run

docker-compose up --build

And access...

curl http://localhost:8080/ping

and I see the Transaction show up...

What am I missing? ... I See the Transactions
I do not see errors in the log.

I am even getting the correlated Logs

And Metrics

[mmarinov (Miroslav Marinov)]

Hi, @stephenb! Thank you for the quick reply!

The APM Server address does end with :443 so it's not the issue.

I was able to narrow down the cause (or at least the exact circumstances) for the issue: All the time I was trying to run the application using IntelliJ, which probably shifts the focus of my initial report a bit. If I build the application and run it outside of the IDE with java -jar ... (like in the docker container from the example) it really does work.

Here is the updated setup:

1. When I try to run a Java application (not even a Spring Boot one) with IntelliJ, instrumented with the OpenTelemetry Agent:
   
   
   
   Environment variables:

OTEL_EXPORTER_OTLP_ENDPOINT=https://xxxx.apm.eu-central-1.aws.cloud.es.io:443;OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer xxxx";OTEL_LOGS_EXPORTER=otlp;OTEL_METRICS_EXPORTER=otlp;OTEL_RESOURCE_ATTRIBUTES=service.name=TimeLoopOTEL,service.version=1.0,deployment.environment=dev

I always get the error from my initial post.

2. However if I run the same application, instrumented with the Elastic APM Agent:

VM Options:

-javaagent:elastic-apm-agent.jar
-Delastic.apm.service_name=TimeLoop
-Delastic.apm.secret_token=xxxx
-Delastic.apm.server_url=https://xxxx.apm.eu-central-1.aws.cloud.es.io:443
-Delastic.apm.environment=dev
-Delastic.apm.application_packages=org.example

Environment variables: none

Everything works fine and I can see the service metrics in Kibana.

The application itself is as simple as possible:

• Main.java

package org.example;

import java.time.Instant;

public class Main {

  public static void main(String[] args) {
    System.out.printf("Press Ctrl+C to quit.");

    while (true) {
      System.out.println("The time is: " + Instant.now().toString());
      try {
        Thread.sleep(5000);
      } catch (InterruptedException e) {
        System.out.println("Sleep was interrupted.");
        throw new RuntimeException(e);
      }
    }
  }
}

• build.gradle

plugins {
    id 'java'
}

group = 'org.example'
version = '1.0-SNAPSHOT'

repositories {
    mavenCentral()
}

dependencies {
    testImplementation platform('org.junit:junit-bom:5.9.1')
    testImplementation 'org.junit.jupiter:junit-jupiter'
}

test {
    useJUnitPlatform()
}

I understand that probably IntelliJ's own agent, that it uses to start the application from the IDE could be the cause of the issue, however it's still strange that it only affects the OTEL agent and does not affect the Elastic APM one.

[mmarinov (Miroslav Marinov)]

One more update: If I add the OpenTelemetry Collector between the OTEL Agent and the APM Server it works.

Here is the summary:

• Running a compiled JAR with java -jar ... -javaagent:... :

  • java -> opentelemetry-javaagent -> APM Server: works
  • java -> elastic-apm-agent -> APM Server: works

• Running the application from IntelliJ:

  • IntelliJ -> opentelemetry-javaagent -> APM Server: doesn't work (PROTCOL_ERROR)
  • IntelliJ -> opentelemetry-javaagent -> OpenTelemetry Collector -> APM Server: works
  • IntelliJ -> elastic-apm-agent -> APM Server: works

Probably one of the major differences with the Collector in place is that the connection to the collector is insecure.

[stephenb (Stephen Brown)]

@mmarinov sorry I can't help with IntelliJ I use VS code these days.

I might go check with the OTEL community or intelliJ community see if anyone knows anything over there.

My one question is are you absolutely sure that you're using the correct and latest open telemetry jar and Java version? In the IntelliJ

[mmarinov (Miroslav Marinov)]

@stephenb The opentelemetry-javaagent and the Java SDK are the same in all test cases and are the latest versions (actually the Java SDK may be several minor versions behind - I will check with the latest version as well).
Thanks for you time!
I will write back if I find a solution.