# Can not export traces from opentelemetry collector to APM Server: request to https://apm.\<removed\>:8200/v1/traces responded with HTTP Status Code 400

**URL:** <https://discuss.elastic.co/t/can-not-export-traces-from-opentelemetry-collector-to-apm-server-request-to-https-apm-removed-8200-v1-traces-responded-with-http-status-code-400/313492>\
**Category:** APM\
**Tags:** docker, open-telemetry\
**Created:** [September 2, 2022, 1:20am UTC](https://discuss.elastic.co/t/can-not-export-traces-from-opentelemetry-collector-to-apm-server-request-to-https-apm-removed-8200-v1-traces-responded-with-http-status-code-400/313492 "2022-09-02T01:20:44Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![ramdas](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ramdas/32/110363_2.png) [@ramdas](https://discuss.elastic.co/u/ramdas)\
**Post date:** [September 2, 2022, 1:20am UTC](https://discuss.elastic.co/t/can-not-export-traces-from-opentelemetry-collector-to-apm-server-request-to-https-apm-removed-8200-v1-traces-responded-with-http-status-code-400/313492/1 "2022-09-02T01:20:44Z")

</div>

Hi,

We have our application instrumented to push traces on otlp to open telemetry collector and we are then exporting traces to elastic APM. we were earlier using "elastic" exporter to push traces to elastic APM server directly and it was working perfectly fine until it was deprecated and we are forced to look for alternatives.

as per latest documentation, we are now trying to use otlp or otlphttp exporter in our opentelemetry collector, however it is not working. latest status is we are getting HTTP 400 while using otlphttp exporter.

I am using ingress with load balancer service type for APM server for SSL termination. opentelemetry collector is using clusterip service type.

Stack :  
Open telemetry collector image: [docker.io/otel/opentelemetry-collector-contrib:latest](http://docker.io/otel/opentelemetry-collector-contrib:latest)  
elastic APM Server: 8.4.1

otel-collector config:

```auto
exporters:
      logging:
        logLevel: debug
      otlphttp/insecure_no_verify:
        endpoint: https://apm.<removed>:8200
        tls:
          insecure: false
          insecure_skip_verify: true
    extensions:
      health_check: {}
      memory_ballast: {}
    processors:
      attributes:
        actions:
        - action: insert
          key: bmap.plan_id
          value: 63075731126d24faff9b56ca
        - action: insert
          key: bmap.plan_name
          value: hj-otel3
      batch: {}
      memory_limiter:
        check_interval: 5s
        limit_mib: 1638
        spike_limit_mib: 512
    receivers:
      otlp:
        protocols:
          grpc:
            endpoint: 0.0.0.0:4317
          http:
            endpoint: 0.0.0.0:4318
    service:
      extensions:
      - health_check
      - memory_ballast
      pipelines:
        traces:
          exporters:
          - logging
          - elastic
          - otlphttp/insecure_no_verify
          processors:
          - attributes
          receivers:
          - otlp
      telemetry:
        metrics:
          address: 0.0.0.0:8888

```

APM Server config:

```auto
    apm-server:
      host: "0.0.0.0:8200"
    ssl:
      enabled: true
      certificate: "/usr/share/apm-server/certs/tls.crt"
      key: "/usr/share/apm-server/certs/tls.key"
      supported_protocols: [TLSv1.1, TLSv1.2, TLSv1.3]
      client_authentication: "none"

    queue: {}
    auth:
      anonymous:
        enabled: true

    output.elasticsearch:
      hosts: ["https://elastic.<removed>:9200"]
      username: "elastic"
      password: "<removed>"

```

APM Server logs:

```auto
{"log.level":"info","@timestamp":"2022-09-02T00:30:10.832Z","log.origin":{"file.name":"instance/beat.go","file.line":702},"message":"Home path: [/usr/share/apm-server] Config path: [/usr/share/apm-server] Data path: [/usr/share/apm-server/data] Logs path: [/usr/share/apm-server/logs]","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-09-02T00:30:10.843Z","log.origin":{"file.name":"instance/beat.go","file.line":710},"message":"Beat ID: e1cd9361-c2cd-4cb2-82c0-8a2906ad5eec","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-09-02T00:30:10.843Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1076},"message":"Beat info","service.name":"apm-server","system_info":{"beat":{"path":{"config":"/usr/share/apm-server","data":"/usr/share/apm-server/data","home":"/usr/share/apm-server","logs":"/usr/share/apm-server/logs"},"type":"apm-server","uuid":"e1cd9361-c2cd-4cb2-82c0-8a2906ad5eec"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-09-02T00:30:10.843Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1085},"message":"Build info","service.name":"apm-server","system_info":{"build":{"commit":"54b929734617c29cf0b5415d4f1e31e5c080b7ad","libbeat":"8.4.0","time":"2022-08-25T18:49:04.000Z","version":"8.4.1"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-09-02T00:30:10.843Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1088},"message":"Go runtime info","service.name":"apm-server","system_info":{"go":{"os":"linux","arch":"amd64","max_procs":4,"version":"go1.17.12"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-09-02T00:30:10.844Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1092},"message":"Host info","service.name":"apm-server","system_info":{"host":{"architecture":"x86_64","boot_time":"2022-08-30T15:09:12Z","containerized":true,"name":"apm-apm-server-8b87fd85-tscs6","ip":["127.0.0.1/8","::1/128","10.12.249.167/21","fe80::5424:72ff:fed0:856e/64"],"kernel_version":"5.4.0-1089-azure","mac":["56:24:72:d0:85:6e"],"os":{"type":"linux","family":"debian","platform":"ubuntu","name":"Ubuntu","version":"20.04.4 LTS (Focal Fossa)","major":20,"minor":4,"patch":4,"codename":"focal"},"timezone":"UTC","timezone_offset_sec":0},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-09-02T00:30:10.844Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1121},"message":"Process info","service.name":"apm-server","system_info":{"process":{"capabilities":{"inheritable":null,"permitted":null,"effective":null,"bounding":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"ambient":null},"cwd":"/usr/share/apm-server","exe":"/usr/share/apm-server/apm-server","name":"apm-server","pid":8,"ppid":1,"seccomp":{"mode":"disabled","no_new_privs":false},"start_time":"2022-09-02T00:30:10.220Z"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-09-02T00:30:10.844Z","log.origin":{"file.name":"instance/beat.go","file.line":293},"message":"Setup Beat: apm-server; Version: 8.4.1","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-09-02T00:30:10.847Z","log.logger":"esclientleg","log.origin":{"file.name":"eslegclient/connection.go","file.line":108},"message":"elasticsearch url: https://elastic.<removed>.com:9200","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-09-02T00:30:10.847Z","log.logger":"publisher","log.origin":{"file.name":"pipeline/module.go","file.line":113},"message":"Beat name: apm-apm-server-8b87fd85-tscs6","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-09-02T00:30:10.848Z","log.origin":{"file.name":"instance/beat.go","file.line":470},"message":"apm-server start running.","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-09-02T00:30:10.848Z","log.logger":"beater","log.origin":{"file.name":"beater/http.go","file.line":144},"message":"Listening on: [::]:8200","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-09-02T00:30:10.848Z","log.logger":"beater","log.origin":{"file.name":"beater/beater.go","file.line":1073},"message":"maxprocs: Updating GOMAXPROCS=1: determined from CPU quota","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-09-02T00:30:10.850Z","log.logger":"beater","log.origin":{"file.name":"beater/waitready.go","file.line":40},"message":"blocking ingestion until all preconditions are satisfied","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-09-02T00:30:10.850Z","log.logger":"beater","log.origin":{"file.name":"apm-server/main.go","file.line":64},"message":"creating transaction metrics aggregation with config: {Interval:1m0s MaxTransactionGroups:10000 HDRHistogramSignificantFigures:2}","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-09-02T00:30:10.853Z","log.logger":"beater","log.origin":{"file.name":"apm-server/main.go","file.line":79},"message":"creating service destinations aggregation with config: {Interval:1m0s MaxGroups:10000}","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-09-02T00:30:10.854Z","log.logger":"handler","log.origin":{"file.name":"api/mux.go","file.line":140},"message":"Path / added to request handler","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-09-02T00:30:10.854Z","log.logger":"handler","log.origin":{"file.name":"api/mux.go","file.line":140},"message":"Path /config/v1/agents added to request handler","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-09-02T00:30:10.854Z","log.logger":"handler","log.origin":{"file.name":"api/mux.go","file.line":140},"message":"Path /config/v1/rum/agents added to request handler","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-09-02T00:30:10.854Z","log.logger":"handler","log.origin":{"file.name":"api/mux.go","file.line":140},"message":"Path /intake/v2/rum/events added to request handler","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-09-02T00:30:10.854Z","log.logger":"handler","log.origin":{"file.name":"api/mux.go","file.line":140},"message":"Path /intake/v3/rum/events added to request handler","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-09-02T00:30:10.854Z","log.logger":"handler","log.origin":{"file.name":"api/mux.go","file.line":140},"message":"Path /intake/v2/events added to request handler","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-09-02T00:30:10.854Z","log.logger":"handler","log.origin":{"file.name":"api/mux.go","file.line":140},"message":"Path /intake/v2/profile added to request handler","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-09-02T00:30:10.854Z","log.logger":"handler","log.origin":{"file.name":"api/mux.go","file.line":140},"message":"Path /v1/traces added to request handler","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-09-02T00:30:10.854Z","log.logger":"handler","log.origin":{"file.name":"api/mux.go","file.line":140},"message":"Path /v1/metrics added to request handler","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-09-02T00:30:10.854Z","log.logger":"handler","log.origin":{"file.name":"api/mux.go","file.line":140},"message":"Path /v1/logs added to request handler","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-09-02T00:30:10.854Z","log.logger":"beater","log.origin":{"file.name":"beater/server.go","file.line":233},"message":"Starting apm-server [54b929734617c29cf0b5415d4f1e31e5c080b7ad built 2022-08-25 18:49:04 +0000 UTC]. Hit CTRL-C to stop it.","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-09-02T00:30:10.854Z","log.logger":"beater","log.origin":{"file.name":"beater/http.go","file.line":96},"message":"RUM endpoints disabled.","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-09-02T00:30:10.854Z","log.logger":"beater","log.origin":{"file.name":"beater/http.go","file.line":106},"message":"SSL disabled.","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-09-02T00:30:46.476Z","log.logger":"request","log.origin":{"file.name":"middleware/log_middleware.go","file.line":61},"message":"request ok","service.name":"apm-server","url.original":"/","http.request.method":"GET","user_agent.original":"kube-probe/1.23","source.address":"10.12.249.148","http.request.body.bytes":0,"http.request.id":"7e3783a4-fb1f-4015-8fcc-dfb825d43a7a","event.duration":102870,"http.response.status_code":200,"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-09-02T00:30:46.477Z","log.logger":"request","log.origin":{"file.name":"middleware/log_middleware.go","file.line":61},"message":"request ok","service.name":"apm-server","url.original":"/","http.request.method":"GET","user_agent.original":"kube-probe/1.23","source.address":"10.12.249.148","http.request.body.bytes":0,"http.request.id":"99203647-0b23-4bb4-bc8b-c85dcbe79fbb","event.duration":46645,"http.response.status_code":200,"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-09-02T00:30:56.476Z","log.logger":"request","log.origin":{"file.name":"middleware/log_middleware.go","file.line":61},"message":"request ok","service.name":"apm-server","url.original":"/","http.request.method":"GET","user_agent.original":"kube-probe/1.23","source.address":"10.12.249.148","http.request.body.bytes":0,"http.request.id":"f4cefe04-9ca3-4662-ae95-b1835fae63bf","event.duration":105186,"http.response.status_code":200,"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-09-02T00:30:56.477Z","log.logger":"request","log.origin":{"file.name":"middleware/log_middleware.go","file.line":61},"message":"request ok","service.name":"apm-server","url.original":"/","http.request.method":"GET","user_agent.original":"kube-probe/1.23","source.address":"10.12.249.148","http.request.body.bytes":0,"http.request.id":"2d7db3f1-1c46-4302-b0bb-9c958b743914","event.duration":64670,"http.response.status_code":200,"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-09-02T00:30:56.871Z","log.logger":"request","log.origin":{"file.name":"middleware/log_middleware.go","file.line":61},"message":"handled request","service.name":"apm-server","url.original":"/v1/traces","http.request.method":"POST","user_agent.original":"OpenTelemetry Collector Contrib/0.56.0 (linux/amd64)","source.address":"10.12.248.56","http.request.body.bytes":813,"http.request.id":"c1e583d0-fcef-45f1-b24e-54870fa856da","event.duration":146745,"http.response.status_code":200,"ecs.version":"1.6.0"}
**{"log.level":"info","@timestamp":"2022-09-02T00:31:03.687Z","log.logger":"request","log.origin":{"file.name":"middleware/log_middleware.go","file.line":61},"message":"handled request","service.name":"apm-server","url.original":"/v1/traces","http.request.method":"POST","user_agent.original":"OpenTelemetry Collector Contrib/0.56.0 (linux/amd64)","source.address":"10.12.248.56","http.request.body.bytes":815,"http.request.id":"ab25dfa0-c6bb-4e8f-ba75-f665678ae221","event.duration":64960,"http.response.status_code":200,"ecs.version":"1.6.0"}**
{"log.level":"info","@timestamp":"2022-09-02T00:31:06.477Z","log.logger":"request","log.origin":{"file.name":"middleware/log_middleware.go","file.line":61},"message":"request ok","service.name":"apm-server","url.original":"/","http.request.method":"GET","user_agent.original":"kube-probe/1.23","source.address":"10.12.249.148","http.request.body.bytes":0,"http.request.id":"2839fceb-1534-483e-96bb-6d7f349fe7a9","event.duration":104900,"http.response.status_code":200,"ecs.version":"1.6.0"}

```

collector logs:

```auto
2022/09/02 00:38:49 proto: duplicate proto type registered: jaeger.api_v2.PostSpansRequest
2022/09/02 00:38:49 proto: duplicate proto type registered: jaeger.api_v2.PostSpansResponse
2022-09-02T00:38:49.125Z info service/telemetry.go:102 Setting up own telemetry...
2022-09-02T00:38:49.126Z info service/telemetry.go:137 Serving Prometheus metrics {"address": "0.0.0.0:8888", "level": "basic"}
2022-09-02T00:38:49.126Z info components/components.go:30 In development component. May change in the future. {"kind": "exporter", "data_type": "traces", "name": "logging", "stability": "in development"}
2022-09-02T00:38:49.126Z info components/components.go:30 Deprecated component. Will be removed in future releases. {"kind": "exporter", "data_type": "traces", "name": "elastic", "stability": "deprecated"}
2022-09-02T00:38:49.126Z warn elasticexporter@v0.56.0/factory.go:53 elastic exporter is deprecated and will be removed in future versions. {"kind": "exporter", "data_type": "traces", "name": "elastic"}
2022-09-02T00:38:49.126Z info extensions/extensions.go:42 Starting extensions...
2022-09-02T00:38:49.126Z info extensions/extensions.go:45 Extension is starting... {"kind": "extension", "name": "memory_ballast"}
2022-09-02T00:38:49.127Z info ballastextension/memory_ballast.go:52 Setting memory ballast {"kind": "extension", "name": "memory_ballast", "MiBs": 0}
2022-09-02T00:38:49.127Z info extensions/extensions.go:49 Extension started. {"kind": "extension", "name": "memory_ballast"}
2022-09-02T00:38:49.127Z info extensions/extensions.go:45 Extension is starting... {"kind": "extension", "name": "health_check"}
2022-09-02T00:38:49.127Z info healthcheckextension@v0.56.0/healthcheckextension.go:44 Starting health_check extension {"kind": "extension", "name": "health_check", "config": {"Port":0,"TCPAddr":{"Endpoint":"0.0.0.0:13133"},"Path":"/","CheckCollectorPipeline":{"Enabled":false,"Interval":"5m","ExporterFailureThreshold":5}}}
2022-09-02T00:38:49.127Z info extensions/extensions.go:49 Extension started. {"kind": "extension", "name": "health_check"}
2022-09-02T00:38:49.127Z info pipelines/pipelines.go:74 Starting exporters...
2022-09-02T00:38:49.127Z info pipelines/pipelines.go:78 Exporter is starting... {"kind": "exporter", "data_type": "traces", "name": "logging"}
2022-09-02T00:38:49.127Z info pipelines/pipelines.go:82 Exporter started. {"kind": "exporter", "data_type": "traces", "name": "logging"}
2022-09-02T00:38:49.127Z info pipelines/pipelines.go:78 Exporter is starting... {"kind": "exporter", "data_type": "traces", "name": "elastic"}
2022-09-02T00:38:49.127Z info pipelines/pipelines.go:82 Exporter started. {"kind": "exporter", "data_type": "traces", "name": "elastic"}
2022-09-02T00:38:49.127Z info pipelines/pipelines.go:78 Exporter is starting... {"kind": "exporter", "data_type": "traces", "name": "otlphttp/insecure_no_verify"}
2022-09-02T00:38:49.127Z info pipelines/pipelines.go:82 Exporter started. {"kind": "exporter", "data_type": "traces", "name": "otlphttp/insecure_no_verify"}
2022-09-02T00:38:49.127Z info pipelines/pipelines.go:86 Starting processors...
2022-09-02T00:38:49.127Z info pipelines/pipelines.go:90 Processor is starting... {"kind": "processor", "name": "attributes", "pipeline": "traces"}
2022-09-02T00:38:49.127Z info pipelines/pipelines.go:94 Processor started. {"kind": "processor", "name": "attributes", "pipeline": "traces"}
2022-09-02T00:38:49.127Z info pipelines/pipelines.go:98 Starting receivers...
2022-09-02T00:38:49.127Z info pipelines/pipelines.go:102 Receiver is starting... {"kind": "receiver", "name": "otlp", "pipeline": "traces"}
2022-09-02T00:38:49.127Z info otlpreceiver/otlp.go:70 Starting GRPC server on endpoint 0.0.0.0:4317 {"kind": "receiver", "name": "otlp", "pipeline": "traces"}
2022-09-02T00:38:49.127Z info otlpreceiver/otlp.go:88 Starting HTTP server on endpoint 0.0.0.0:4318 {"kind": "receiver", "name": "otlp", "pipeline": "traces"}
2022-09-02T00:38:49.127Z info pipelines/pipelines.go:106 Receiver started. {"kind": "receiver", "name": "otlp", "pipeline": "traces"}
2022-09-02T00:38:49.127Z info healthcheck/handler.go:129 Health Check state change {"kind": "extension", "name": "health_check", "status": "ready"}
2022-09-02T00:38:49.127Z info service/collector.go:215 Starting otelcol-contrib... {"Version": "0.56.0", "NumCPU": 4}
2022-09-02T00:38:49.127Z info service/collector.go:128 Everything is ready. Begin running and processing data.
2022-09-02T00:41:35.755Z info TracesExporter {"kind": "exporter", "data_type": "traces", "name": "logging", "#spans": 5}
2022-09-02T00:41:35.755Z info ResourceSpans #0
Resource SchemaURL:
Resource labels:
     -> service.name: STRING(PWDI Server)
     -> service.instance.id: STRING(6bc70ae6-f0c0-44da-9ef6-07eef3f2b2f8)
ScopeSpans #0
ScopeSpans SchemaURL:
InstrumentationScope PWDI
Span #0
    Trace ID : 7b12a75510ff3c4ebc59130bcefa7615
    Parent ID :
    ID : 14ad637bb8b33a40
    Name : PwDbOperation
    Kind : SPAN_KIND_SERVER
    Start time : 2022-09-02 00:41:31.6457119 +0000 UTC
    End time : 2022-09-02 00:41:31.649067 +0000 UTC
    Status code : STATUS_CODE_UNSET
    Status message :
Attributes:
     -> session: STRING(0xb1ea66e0)
     -> dbAction: STRING(ExecuteSelect)
     -> sqlCmd: STRING(select schema_name())
     -> datasource: STRING(pwdsn_db0)
     -> bmap.plan_id: STRING(63075731126d24faff9b56ca)
     -> bmap.plan_name: STRING(hj-otel3)
Span #1
    Trace ID : c474cb5be6aad542829d6afc5f8cd85f
    Parent ID :
    ID : de61c58f5cf0c345
    Name : PwDbOperation
    Kind : SPAN_KIND_SERVER
    Start time : 2022-09-02 00:41:31.7186582 +0000 UTC
    End time : 2022-09-02 00:41:31.7221124 +0000 UTC
    Status code : STATUS_CODE_UNSET
    Status message :
Attributes:
     -> session: STRING(0xb1ea66e0)
     -> dbAction: STRING(ExecuteSelect)
     -> sqlCmd: STRING(select count(*) where ('A' <> 'a'))
     -> datasource: STRING(pwdsn_db0)
     -> bmap.plan_id: STRING(63075731126d24faff9b56ca)
     -> bmap.plan_name: STRING(hj-otel3)
Span #2
    Trace ID : 6b23cbb675479e4b84bb6567d2836115
    Parent ID :
    ID : 5d91048f7506414a
    Name : PwDbOperation
    Kind : SPAN_KIND_SERVER
    Start time : 2022-09-02 00:41:31.7225876 +0000 UTC
    End time : 2022-09-02 00:41:31.7255151 +0000 UTC
    Status code : STATUS_CODE_UNSET
    Status message :
Attributes:
     -> session: STRING(0xb1ea66e0)
     -> dbAction: STRING(ExecuteSelect)
     -> sqlCmd: STRING(select cast(SERVERPROPERTY('EngineEdition') as int))
     -> datasource: STRING(pwdsn_db0)
     -> bmap.plan_id: STRING(63075731126d24faff9b56ca)
     -> bmap.plan_name: STRING(hj-otel3)
Span #3
    Trace ID : ffc7de2ff1eeec49884d7d9803597217
    Parent ID :
    ID : d227094270ed3244
    Name : PwDbOperation
    Kind : SPAN_KIND_SERVER
    Start time : 2022-09-02 00:41:31.7387606 +0000 UTC
    End time : 2022-09-02 00:41:31.7417716 +0000 UTC
    Status code : STATUS_CODE_UNSET
    Status message :
Attributes:
     -> session: STRING(0xb1ea66e0)
     -> dbAction: STRING(ExecuteSelect)
     -> sqlCmd: STRING(select o_verno,o_rectype,o_release,o_major,o_minor,o_build,o_updatetime,o_text from dms_ver where o_rectype = 10 order by o_verno)
     -> datasource: STRING(pwdsn_db0)
     -> bmap.plan_id: STRING(63075731126d24faff9b56ca)
     -> bmap.plan_name: STRING(hj-otel3)
Span #4
    Trace ID : 7033875c3332ed4698f08c6691073f95
    Parent ID :
    ID : 6a6a0e1ca26b8e48
    Name : PwDbOperation
    Kind : SPAN_KIND_SERVER
    Start time : 2022-09-02 00:41:31.8091887 +0000 UTC
    End time : 2022-09-02 00:41:31.8125511 +0000 UTC
    Status code : STATUS_CODE_UNSET
    Status message :
Attributes:
     -> session: STRING(0xb1ea66e0)
     -> dbAction: STRING(ExecuteSelect)
     -> sqlCmd: STRING(select INDEXDOCINFOID,IMPORT from EC_INDEXDOCINFO where (DOC_GUID not in (select o_docguid from dms_doc)) order by IMPORT)
     -> datasource: STRING(pwdsn_db0)
     -> bmap.plan_id: STRING(63075731126d24faff9b56ca)
     -> bmap.plan_name: STRING(hj-otel3)
        {"kind": "exporter", "data_type": "traces", "name": "logging"}
2022-09-02T00:41:35.812Z error exporterhelper/queued_retry.go:183 **Exporting failed. The error is not retryable. Dropping data. {"kind": "exporter", "data_type": "traces", "name": "otlphttp/insecure_no_verify", "error": "Permanent error: error exporting items, request to https://apm.<removed>:8200/v1/traces responded with HTTP Status Code 400, Message=proto: illegal wireType 7, Details=[]", "dropped_items": 5**}
go.opentelemetry.io/collector/exporter/exporterhelper.(*retrySender).send
        go.opentelemetry.io/collector@v0.56.0/exporter/exporterhelper/queued_retry.go:183
go.opentelemetry.io/collector/exporter/exporterhelper.(*tracesExporterWithObservability).send
        go.opentelemetry.io/collector@v0.56.0/exporter/exporterhelper/traces.go:134
go.opentelemetry.io/collector/exporter/exporterhelper.(*queuedRetrySender).start.func1
        go.opentelemetry.io/collector@v0.56.0/exporter/exporterhelper/queued_retry_inmemory.go:119
go.opentelemetry.io/collector/exporter/exporterhelper/internal.consumerFunc.consume
        go.opentelemetry.io/collector@v0.56.0/exporter/exporterhelper/internal/bounded_memory_queue.go:82
go.opentelemetry.io/collector/exporter/exporterhelper/internal.(*boundedMemoryQueue).StartConsumers.func2
        go.opentelemetry.io/collector@v0.56.0/exporter/exporterhelper/internal/bounded_memory_queue.go:69
2022-09-02T00:43:25.011Z info TracesExporter {"kind": "exporter", "data_type": "traces", "name": "logging", "#spans": 5}
2022-09-02T00:43:25.011Z info ResourceSpans #0

```

---

<div class="post-metadata">

**Author:** ![ramdas](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ramdas/32/110363_2.png) [@ramdas](https://discuss.elastic.co/u/ramdas)\
**Post date:** [September 2, 2022, 8:52am UTC](https://discuss.elastic.co/t/can-not-export-traces-from-opentelemetry-collector-to-apm-server-request-to-https-apm-removed-8200-v1-traces-responded-with-http-status-code-400/313492/2 "2022-09-02T08:52:05Z")

</div>

continuing to above post:

if i use otlp exporter instead of otlphttp in collector config,

```auto
otlp/insecure:
        endpoint: https://apm.<removed>:8200
        tls:
          insecure: true

```

then I get HTTP 404

```auto
2022-09-02T08:48:48.433Z error exporterhelper/queued_retry.go:183 Exporting failed. The error is not retryable. Dropping data. {"kind": "exporter", "data_type": "traces", "name": "otlp/insecure", "error": "Permanent error: rpc error: code = Unimplemented desc = unexpected HTTP status code received from server: 404 (Not Found); transport: received unexpected content-type \"text/plain; charset=utf-8\"", "dropped_items": 5}

```

if I use above exporter config without https:// prefix as given in [elastic-apm](https://www.elastic.co/guide/en/apm/guide/current/open-telemetry.html#connect-open-telemetry-collector) docs,

```auto
otlp/insecure:
        endpoint: apm.<removed>:8200
        tls:
          insecure: true

```

then I get below error:

```auto
{"kind": "exporter", "data_type": "traces", "name": "logging"}
2022-09-02T08:53:51.584Z info exporterhelper/queued_retry.go:215 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "traces", "name": "otlp/insecure", "error": "rpc error: code = Unavailable desc = connection closed before server preface received", "interval": "4.055431468s"}

```

---

<div class="post-metadata">

**Author:** ![axw](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/axw/32/28197_2.png) [@axw](https://discuss.elastic.co/u/axw)\
**Post date:** [September 5, 2022, 2:45am UTC](https://discuss.elastic.co/t/can-not-export-traces-from-opentelemetry-collector-to-apm-server-request-to-https-apm-removed-8200-v1-traces-responded-with-http-status-code-400/313492/3 "2022-09-05T02:45:46Z")

</div>

> [@ramdas](#):
>
> as per latest documentation, we are now trying to use otlp or otlphttp exporter in our opentelemetry collector, however it is not working. latest status is we are getting HTTP 400 while using otlphttp exporter.

It appears that we're not handling the otel-collector's default compression properly: [OTLP/HTTP requests fail on compressed payloads · Issue #9036 · elastic/apm-server · GitHub](https://github.com/elastic/apm-server/issues/9036)

Until that is fixed, you can work around this by adding `compression: none` in the `otlphttp` config block.

---

<div class="post-metadata">

**Author:** ![ramdas](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ramdas/32/110363_2.png) [@ramdas](https://discuss.elastic.co/u/ramdas)\
**Post date:** [September 6, 2022, 9:11am UTC](https://discuss.elastic.co/t/can-not-export-traces-from-opentelemetry-collector-to-apm-server-request-to-https-apm-removed-8200-v1-traces-responded-with-http-status-code-400/313492/4 "2022-09-06T09:11:07Z")

</div>

> [@axw](#):
>
> Until that is fixed, you can work around this by adding `compression: none` in the `otlphttp` config block.

Thanks a lot @axw this fixed my issue 🙂

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [September 27, 2022, 5:11am UTC](https://discuss.elastic.co/t/can-not-export-traces-from-opentelemetry-collector-to-apm-server-request-to-https-apm-removed-8200-v1-traces-responded-with-http-status-code-400/313492/5 "2022-09-27T05:11:08Z")

</div>

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.