EDOT pushing to elastic fails with HTTP status message: Bad Request

[yev]

Hi there,
I'm exploring the capabilities of EDOT and I have an issue:

Logs from my spring-boot app

[otel.javaagent 2025-03-10 16:31:10:565 +0100] [OkHttp http://localhost:9200/...] WARN io.opentelemetry.exporter.internal.http.HttpExporter - Failed to export logs. Server responded with HTTP status code 400. Error message: Unable to parse response body, HTTP status message: Bad Request
[otel.javaagent 2025-03-10 16:31:10:571 +0100] [OkHttp TaskRunner] DEBUG okhttp3.internal.concurrent.TaskRunner - Q10004 starting              : OkHttp ConnectionPool
[otel.javaagent 2025-03-10 16:31:10:571 +0100] [BatchLogRecordProcessor_WorkerThread-1] DEBUG io.opentelemetry.sdk.logs.export.BatchLogRecordProcessor$Worker - Exporter failed
[otel.javaagent 2025-03-10 16:31:10:571 +0100] [OkHttp ConnectionPool] DEBUG okhttp3.internal.concurrent.TaskRunner - Q10004 run again after 300 s : OkHttp ConnectionPool
[otel.javaagent 2025-03-10 16:31:10:571 +0100] [OkHttp TaskRunner] DEBUG okhttp3.internal.concurrent.TaskRunner - Q10004 finished run in 251 µs: OkHttp ConnectionPool

Can I push directly to Elastic ? Or I need to pass through some collector? I followed the documentation on gitHub and I thought that I can push data directly to ES...

I'm using Java23, SpringBoot 3.4, latest elastic docker image.

Thank you in advance for help!

[stephenb (Stephen Brown)]

Hi @yev Welcome to the community. And thanks for trying EDOT.

Here are the current valid paths... this is for Java .. not all language SDK may support the APM Server path.

JAVA App -> EDOT Java SDK OLTP exporter -> Elastic APM Server -> Elasticsearch 

JAVA App -> EDOT Java SDK OLTP exporter -> EDOT Collector (exporters.elasticsearch) > Elasticsearch 

Let us know if that makes sense...

[yev]

hi, @stephenb , thank your for your fast answer! I will try!

[stephenb (Stephen Brown)]

It is a fast moving time right now

[yev]

thank you @stephenb

With otelcol-contrib docker I have successfully pushed the logs to my ES.

But now I have another issue with metrics. I enabled metrics in oltp-collector config:

service:
    pipelines:
        logs:
            receivers: [ otlp ]
            exporters: [ elasticsearch ]
        traces:
            receivers: [ otlp ]
            exporters: [ elasticsearch ]
        metrics:
            receivers: [ otlp ]
            exporters: [ elasticsearch ]

And now in collectors logs I see those errors:

2025-03-10T19:22:59.954Z        info    service@v0.121.0/service.go:281 Everything is ready. Begin running and processing data.
2025-03-10T19:23:52.721Z        warn    elasticsearchexporter@v0.121.0/exporter.go:322  validation errors       {"otelcol.component.id": "elasticsearch", "otelcol.component.kind": "Exporter", "otelcol.signal": "metrics", "error": "dropping cumulative temporality histogram \"jvm.gc.duration\"\ndropping cumulative temporality histogram \"http.server.request.duration\"\ndropping cumulative temporality histogram \"db.client.connections.use_time\"\ndropping cumulative temporality histogram \"db.client.connections.wait_time\"\ndropping cumulative temporality histogram \"db.client.connections.create_time\""}
2025-03-10T19:24:52.724Z        warn    elasticsearchexporter@v0.121.0/exporter.go:322  validation errors       {"otelcol.component.id": "elasticsearch", "otelcol.component.kind": "Exporter", "otelcol.signal": "metrics", "error": "dropping cumulative temporality histogram \"jvm.gc.duration\"\ndropping cumulative temporality histogram \"http.server.request.duration\"\ndropping cumulative temporality histogram \"db.client.connections.use_time\"\ndropping cumulative temporality histogram \"db.client.connections.wait_time\"\ndropping cumulative temporality histogram \"db.client.connections.create_time\""}
2025-03-10T19:25:52.721Z        warn    elasticsearchexporter@v0.121.0/exporter.go:322  validation errors       {"otelcol.component.id": "elasticsearch", "otelcol.component.kind": "Exporter", "otelcol.signal": "metrics", "error": "dropping cumulative temporality histogram \"jvm.gc.duration\"\ndropping cumulative temporality histogram \"http.server.request.duration\"\ndropping cumulative temporality histogram \"db.client.connections.use_time\"\ndropping cumulative temporality histogram \"db.client.connections.wait_time\"\ndropping cumulative temporality histogram \"db.client.connections.create_time\""}

Any idea how to fix it?

[stephenb (Stephen Brown)]

Opens a new topic with Proper Subject Line... I suspect there are some specific data type settings... that and I am not sure we support temporality yet.

[adminunix]

Hello @stephenb
I am a little bit confused about valid paths. This github doc indicates that APM Server should not be used in the guideline section:

That guide is correct?

Thank you very much!

[stephenb (Stephen Brown)]

It is really just an artifact of how we see the future.

I think our docs have shifted to actively discouraging sending OTel data through APM server and this is just a natural extension of that

So yes, This should still be valid .. as part of a migration
B) OTEL SDK (otlp) -> APM Server -> ECH (Now, but Elastic APM Format)

[adminunix]

We have a self-managed Elasticsearch cluster so I understand we can use this path and is fully supported:

JAVA App (tomcat) -> EDOT Java  -> Elastic APM Server -> Self-Managed Elasticsearch

On the other hand, with OTEL Collector (gateway mode) I understand that we should use exporters.elasticsearch, APM Connector and APM processor to enrich traces and use APM UIs in kibana?

Thanks!

[stephenb (Stephen Brown)]

Hi @adminunix I am not sure what the question is.
Perhaps you should open a thread (working in an answered thread is not ideal) , with all the details,m Include the versions, etc and what you are trying to accomplish instead of piecemeal.
There are many valid architectures it depends on what your constraints are and what you are trying to accomplish

[yesinmorse]

The HTTP 400 Bad Request error occurs because the EDOT Java SDK cannot push OTLP data directly to the Elasticsearch port (9200), as Elasticsearch requires a specific translation layer to understand OpenTelemetry protocol buffers. To resolve this, you must route your traffic through either the Elastic APM Server or an OpenTelemetry Collector configured with the elasticsearch exporter, which handles the necessary data mapping and indexing.

[stephenb (Stephen Brown)]

@adminunix

JAVA App (tomcat) -> EDOT Java -> Elastic APM Server -> Self-Managed Elasticsearch

Correct EDOT JAVA SDK can not write directly to Elasticsearch you need to either use a the APM Server or OTEL / EDOT Collector

Here are all the permutations

A) Legacy APM (v2) -> APM Server -> ES Self Managed / ECH (Now, Elastic APM Format)

B) OTEL SDK (otlp) -> APM Server -> ES Self Managed /ECH (Now, but Elastic APM Format)

C) EDOT SDK (otlp) -> APM Server -> ES Self Managed /ECH (Now, but Elastic APM Format)

D) Legacy APM (v2) -> EDOT (v2 Reciever) / MOTEL -> ECH (Future) Elastic APM Format

E) EDOT / OTEL SDK (otlp) -> EDOT Collector / OTEL Col -> ES Self Managed / ECH (Now, OTEL Native Schema)

Elastic Cloud Only
F) EDOT / OTEL SDK (otlp)-> mOTLP-> ECH (Future, OTEL Native Schema)
mOTLP is the managed OLTP endpoint. We are aiming for mOTLP for ECH Tech Preview Soon...

G) EDOT / OTEL SDK (otlp)> EDOT / OTEL Col -> mOTLP Col - ECH (Future, OTEL Native Schema) )
mOTLP is the managed OLTP endpoint. We are aiming for mOTLP for ECH Tech Preview Soon...

[adminunix]

Thank you very much Stephen!
I am ready to configure the agents after the info you provided.