# 404 error while pushing traces from OTel Exporter to APM server
**URL:** https://discuss.elastic.co/t/404-error-while-pushing-traces-from-otel-exporter-to-apm-server/315795
**Category:** APM
**Tags:** go, server, open-telemetry
**Created:** [October 4, 2022, 3:00pm UTC](https://discuss.elastic.co/t/404-error-while-pushing-traces-from-otel-exporter-to-apm-server/315795 "2022-10-04T15:00:09Z")
**Posts on this page:** 16
**Page:** 1
### Author:  [@gvencadze](https://discuss.elastic.co/u/gvencadze)
#### Post date: [October 4, 2022, 3:00pm UTC](https://discuss.elastic.co/t/404-error-while-pushing-traces-from-otel-exporter-to-apm-server/315795/1 "2022-10-04T15:00:09Z")
Hey guys, I've setup opentelemetry exporter in cluster, then connected services to it and see traces in exporter logs. But, when they are pushing to APM server it returns `http 404`
```auto
Exporting failed. The error is not retryable. Dropping data. {"kind": "exporter", "data_type": "traces", "name": "otlp", "error": "Permanent error: rpc error: code = Unimplemented desc = unexpected HTTP status code received from server: 404 (Not Found); transport: received unexpected content-type \"text/plain; charset=utf-8\"", "dropped_items": 8}
```
also, I've tried to send data over otlphttp, but got same result:
```auto
Exporting failed. The error is not retryable. Dropping data. {"kind": "exporter", "data_type": "traces", "name": "otlphttp", "error": "Permanent error: error exporting items, request to https://apm-server.example.com:443/v1/traces responded with HTTP Status Code 404", "dropped_items": 8}
```
I've searched a while and found some solutions with enabling insecure mode and compression, tried them, but without success.
We're running APM server v7.17, and same version at local machine with exporter and server is accepting traces as it should be. What I could do with this and how to debug?
---
### Author:  [@tbekiares](https://discuss.elastic.co/u/tbekiares)
#### Post date: [October 4, 2022, 3:14pm UTC](https://discuss.elastic.co/t/404-error-while-pushing-traces-from-otel-exporter-to-apm-server/315795/2 "2022-10-04T15:14:20Z")
Hi Konstantin, can you post the relevant section (exporter) of your OTEL collector configuration?
---
### Author:  [@gvencadze](https://discuss.elastic.co/u/gvencadze)
#### Post date: [October 4, 2022, 3:25pm UTC](https://discuss.elastic.co/t/404-error-while-pushing-traces-from-otel-exporter-to-apm-server/315795/3 "2022-10-04T15:25:54Z")
@tbekiares sure, here it is:
```yaml
config:
receivers:
jaeger: null
zipkin: null
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
http:
endpoint: 0.0.0.0:4318
prometheus: null
processors:
batch: {}
memory_limiter: null
extensions:
health_check: {}
exporters:
logging:
loglevel: debug # debug level is only for stage, do not use in production if unnecessary
otlp:
compression: none
endpoint: ${ELASTIC_APM_URL}
tls:
insecure: true
insecure_skip_verify: true
headers:
Authorization: Bearer ${ELASTIC_APM_SERVER_TOKEN}
otlphttp:
compression: none
endpoint: ${ELASTIC_APM_URL}
tls:
insecure: true
headers:
Authorization: Bearer ${ELASTIC_APM_SERVER_TOKEN}
service:
extensions:
- health_check
telemetry:
metrics:
address: ":8888"
pipelines:
traces:
receivers: [otlp]
processors: [batch]
exporters: [otlp,otlphttp,logging]
metrics: null
logs: null
```
---
### Author:  [@marclop](https://discuss.elastic.co/u/marclop)
#### Post date: [October 5, 2022, 2:47am UTC](https://discuss.elastic.co/t/404-error-while-pushing-traces-from-otel-exporter-to-apm-server/315795/4 "2022-10-05T02:47:23Z")
Hi @gvencadze,
It looks like you folks are using the OTLP HTTP transport in the exporter as well as the OTLP GRCP. APM Server only supports OTLP GRPC up until version `8.3.0`. [APM version 8.3 | APM User Guide [8.4] | Elastic](https://www.elastic.co/guide/en/apm/guide/current/release-notes-8.3.html#_added_2).
The error logs seem to indicate that the HTTP endpoint returns a 404, as you would expect since OTLP over HTTP isn't supported in `7.17.x`.
Try only using the `otlp` exporter and see if that fixes the issue.
---
### Author:  [@gvencadze](https://discuss.elastic.co/u/gvencadze)
#### Post date: [October 5, 2022, 6:32am UTC](https://discuss.elastic.co/t/404-error-while-pushing-traces-from-otel-exporter-to-apm-server/315795/5 "2022-10-05T06:32:41Z")
@marclop hi, we’ve already tried to send traces only over grpc and result was the same — 404
---
### Author:  [@tbekiares](https://discuss.elastic.co/u/tbekiares)
#### Post date: [October 5, 2022, 7:11pm UTC](https://discuss.elastic.co/t/404-error-while-pushing-traces-from-otel-exporter-to-apm-server/315795/6 "2022-10-05T19:11:00Z")
Hi Konstantin,
what version of Elasticsearch are you running? Is this a self-managed cluster? or an Elastic Cloud cluister? What does your ${ELASTIC\_APM\_URL} look like? For otlp/grpc, it should be just apmEndpoint:8200 (or 443, depending on your deployment), no http:// or https:// prefix. Drop 'compression: none' from otlp, and drop the whole 'otlphttp' block.
---
### Author:  [@gvencadze](https://discuss.elastic.co/u/gvencadze)
#### Post date: [October 5, 2022, 9:02pm UTC](https://discuss.elastic.co/t/404-error-while-pushing-traces-from-otel-exporter-to-apm-server/315795/7 "2022-10-05T21:02:01Z")
Hi, we are running v7.17.4 on docker swarm (idk why).
I forgot to remove prefix https from apm endpoint. Also I've removed compression and drop 'otlphttp'
Now I've got new error:
```auto
rpc error: code = Unavailable desc = connection closed before server preface received
```
---
### Author:  [@tbekiares](https://discuss.elastic.co/u/tbekiares)
#### Post date: [October 6, 2022, 10:55am UTC](https://discuss.elastic.co/t/404-error-while-pushing-traces-from-otel-exporter-to-apm-server/315795/8 "2022-10-06T10:55:37Z")
Hi, just to double-check, you are sending specifically to your APM server (and not an elasticsearch node)? And your APM server's port is being properly exposed from the docker container (usually 8200 or 443)?
---
### Author:  [@gvencadze](https://discuss.elastic.co/u/gvencadze)
#### Post date: [October 6, 2022, 10:56am UTC](https://discuss.elastic.co/t/404-error-while-pushing-traces-from-otel-exporter-to-apm-server/315795/9 "2022-10-06T10:56:58Z")
Hi, yeah, we have a nginx proxy that forwards 443 to 3 docker containers with exposed 8200 port
---
### Author:  [@tbekiares](https://discuss.elastic.co/u/tbekiares)
#### Post date: [October 6, 2022, 11:09am UTC](https://discuss.elastic.co/t/404-error-while-pushing-traces-from-otel-exporter-to-apm-server/315795/10 "2022-10-06T11:09:23Z")
and you have nginx setup to proxy http2 (grpc requires http2), not just http1.1?
---
### Author:  [@tbekiares](https://discuss.elastic.co/u/tbekiares)
#### Post date: [October 6, 2022, 11:10am UTC](https://discuss.elastic.co/t/404-error-while-pushing-traces-from-otel-exporter-to-apm-server/315795/11 "2022-10-06T11:10:22Z")
something like `listen 443 ssl http2 default_server;` ?
---
### Author:  [@tbekiares](https://discuss.elastic.co/u/tbekiares)
#### Post date: [October 6, 2022, 11:12am UTC](https://discuss.elastic.co/t/404-error-while-pushing-traces-from-otel-exporter-to-apm-server/315795/12 "2022-10-06T11:12:07Z")
hmm.. does nginx actually support http2 reverse proxy...? i'm not sure.
---
### Author:  [@gvencadze](https://discuss.elastic.co/u/gvencadze)
#### Post date: [October 6, 2022, 11:12am UTC](https://discuss.elastic.co/t/404-error-while-pushing-traces-from-otel-exporter-to-apm-server/315795/13 "2022-10-06T11:12:28Z")
I'll ask our sysadmins and return with result
---
### Author:  [@gvencadze](https://discuss.elastic.co/u/gvencadze)
#### Post date: [October 6, 2022, 3:04pm UTC](https://discuss.elastic.co/t/404-error-while-pushing-traces-from-otel-exporter-to-apm-server/315795/14 "2022-10-06T15:04:07Z")
We've enabled http2 on nginx, but result is the same. We look at logs and didn't see any request to `/v1/traces` as written in OTel documentation. Is this endpoint correct or it's converting into `/intake/v2/events`?
Here is our logs:
```auto
2022/10/06 17:50:11 [error] 992832#992832: *9961 client intended to send too large body: 9534672 bytes, client: 10.10.16.32, server: apm-server., request: "POST /intake/v2/events HTTP/2.0", host: "apm-server."
2022/10/06 17:50:37 [error] 992832#992832: *10521 client intended to send too large body: 4353980 bytes, client: 10.10.16.32, server: apm-server., request: "POST /intake/v2/events HTTP/2.0", host: "apm-server."
```
UPD: we find possible problem in our nginx config, I'll write about results later
---
### Author:  [@gvencadze](https://discuss.elastic.co/u/gvencadze)
#### Post date: [October 25, 2022, 9:57pm UTC](https://discuss.elastic.co/t/404-error-while-pushing-traces-from-otel-exporter-to-apm-server/315795/15 "2022-10-25T21:57:06Z")
For those who will find this topic with same problem:
I couldn't make it work, but I think in my case problem is somewhere on nginx and docker swarm, not in APM server itself
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [November 15, 2022, 5:57pm UTC](https://discuss.elastic.co/t/404-error-while-pushing-traces-from-otel-exporter-to-apm-server/315795/16 "2022-11-15T17:57:47Z")
This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.