404 error while pushing traces from OTel Exporter to APM server

Elastic Observability APM
, ,
1

Hey guys, I've setup opentelemetry exporter in cluster, then connected services to it and see traces in exporter logs. But, when they are pushing to APM server it returns http 404

Exporting failed. The error is not retryable. Dropping data.	{"kind": "exporter", "data_type": "traces", "name": "otlp", "error": "Permanent error: rpc error: code = Unimplemented desc = unexpected HTTP status code received from server: 404 (Not Found); transport: received unexpected content-type \"text/plain; charset=utf-8\"", "dropped_items": 8}

also, I've tried to send data over otlphttp, but got same result:

Exporting failed. The error is not retryable. Dropping data.	{"kind": "exporter", "data_type": "traces", "name": "otlphttp", "error": "Permanent error: error exporting items, request to https://apm-server.example.com:443/v1/traces responded with HTTP Status Code 404", "dropped_items": 8}

I've searched a while and found some solutions with enabling insecure mode and compression, tried them, but without success.

We're running APM server v7.17, and same version at local machine with exporter and server is accepting traces as it should be. What I could do with this and how to debug?

2

Hi Konstantin, can you post the relevant section (exporter) of your OTEL collector configuration?

3

@tbekiares sure, here it is:

config:
  receivers:
    jaeger: null
    zipkin: null
    otlp:
      protocols:
        grpc:
          endpoint: 0.0.0.0:4317
        http:
          endpoint: 0.0.0.0:4318
    prometheus: null

  processors:
    batch: {}
    memory_limiter: null

  extensions:
    health_check: {}

  exporters:
    logging:
      loglevel: debug # debug level is only for stage, do not use in production if unnecessary
    otlp:
      compression: none
      endpoint: ${ELASTIC_APM_URL}
      tls:
        insecure: true
        insecure_skip_verify: true
      headers:
        Authorization: Bearer ${ELASTIC_APM_SERVER_TOKEN}
    otlphttp:
      compression: none
      endpoint: ${ELASTIC_APM_URL}
      tls:
        insecure: true
      headers:
        Authorization: Bearer ${ELASTIC_APM_SERVER_TOKEN}

  service:
    extensions:
      - health_check
    telemetry:
      metrics:
        address: ":8888"

    pipelines:
      traces:
        receivers: [otlp]
        processors: [batch]
        exporters: [otlp,otlphttp,logging]
      metrics: null
      logs: null
4

Hi @gvencadze,

It looks like you folks are using the OTLP HTTP transport in the exporter as well as the OTLP GRCP. APM Server only supports OTLP GRPC up until version 8.3.0. APM version 8.3 | APM User Guide [8.4] | Elastic.

The error logs seem to indicate that the HTTP endpoint returns a 404, as you would expect since OTLP over HTTP isn't supported in 7.17.x.

Try only using the otlp exporter and see if that fixes the issue.

5

@marclop hi, we’ve already tried to send traces only over grpc and result was the same — 404

6

Hi Konstantin,

what version of Elasticsearch are you running? Is this a self-managed cluster? or an Elastic Cloud cluister? What does your ${ELASTIC_APM_URL} look like? For otlp/grpc, it should be just apmEndpoint:8200 (or 443, depending on your deployment), no http:// or https:// prefix. Drop 'compression: none' from otlp, and drop the whole 'otlphttp' block.

7

Hi, we are running v7.17.4 on docker swarm (idk why).

I forgot to remove prefix https from apm endpoint. Also I've removed compression and drop 'otlphttp'

Now I've got new error:

rpc error: code = Unavailable desc = connection closed before server preface received
8

Hi, just to double-check, you are sending specifically to your APM server (and not an elasticsearch node)? And your APM server's port is being properly exposed from the docker container (usually 8200 or 443)?

9

Hi, yeah, we have a nginx proxy that forwards 443 to 3 docker containers with exposed 8200 port

10

and you have nginx setup to proxy http2 (grpc requires http2), not just http1.1?

11

something like listen 443 ssl http2 default_server; ?

12

hmm.. does nginx actually support http2 reverse proxy...? i'm not sure.

13

I'll ask our sysadmins and return with result

14

We've enabled http2 on nginx, but result is the same. We look at logs and didn't see any request to /v1/traces as written in OTel documentation. Is this endpoint correct or it's converting into /intake/v2/events?

Here is our logs:

2022/10/06 17:50:11 [error] 992832#992832: *9961 client intended to send too large body: 9534672 bytes, client: 10.10.16.32, server: apm-server.<removed>, request: "POST /intake/v2/events HTTP/2.0", host: "apm-server.<removed>"
2022/10/06 17:50:37 [error] 992832#992832: *10521 client intended to send too large body: 4353980 bytes, client: 10.10.16.32, server: apm-server.<removed>, request: "POST /intake/v2/events HTTP/2.0", host: "apm-server.<removed>"

UPD: we find possible problem in our nginx config, I'll write about results later

15

For those who will find this topic with same problem:
I couldn't make it work, but I think in my case problem is somewhere on nginx and docker swarm, not in APM server itself

16

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.