Can not export traces from opentelemetry collector to APM Server: request to https://apm.<removed>:8200/v1/traces responded with HTTP Status Code 400

Hi,

We have our application instrumented to push traces on otlp to open telemetry collector and we are then exporting traces to elastic APM. we were earlier using "elastic" exporter to push traces to elastic APM server directly and it was working perfectly fine until it was deprecated and we are forced to look for alternatives.

as per latest documentation, we are now trying to use otlp or otlphttp exporter in our opentelemetry collector, however it is not working. latest status is we are getting HTTP 400 while using otlphttp exporter.

I am using ingress with load balancer service type for APM server for SSL termination. opentelemetry collector is using clusterip service type.

Stack :
Open telemetry collector image: docker.io/otel/opentelemetry-collector-contrib:latest
elastic APM Server: 8.4.1

otel-collector config:

exporters:
      logging:
        logLevel: debug
      otlphttp/insecure_no_verify:
        endpoint: https://apm.<removed>:8200
        tls:
          insecure: false
          insecure_skip_verify: true
    extensions:
      health_check: {}
      memory_ballast: {}
    processors:
      attributes:
        actions:
        - action: insert
          key: bmap.plan_id
          value: 63075731126d24faff9b56ca
        - action: insert
          key: bmap.plan_name
          value: hj-otel3
      batch: {}
      memory_limiter:
        check_interval: 5s
        limit_mib: 1638
        spike_limit_mib: 512
    receivers:
      otlp:
        protocols:
          grpc:
            endpoint: 0.0.0.0:4317
          http:
            endpoint: 0.0.0.0:4318
    service:
      extensions:
      - health_check
      - memory_ballast
      pipelines:
        traces:
          exporters:
          - logging
          - elastic
          - otlphttp/insecure_no_verify
          processors:
          - attributes
          receivers:
          - otlp
      telemetry:
        metrics:
          address: 0.0.0.0:8888

APM Server config:

    apm-server:
      host: "0.0.0.0:8200"
    ssl:
      enabled: true
      certificate: "/usr/share/apm-server/certs/tls.crt"
      key: "/usr/share/apm-server/certs/tls.key"
      supported_protocols: [TLSv1.1, TLSv1.2, TLSv1.3]
      client_authentication: "none"

    queue: {}
    auth:
      anonymous:
        enabled: true

    output.elasticsearch:
      hosts: ["https://elastic.<removed>:9200"]
      username: "elastic"
      password: "<removed>"

APM Server logs:

{"log.level":"info","@timestamp":"2022-09-02T00:30:10.832Z","log.origin":{"file.name":"instance/beat.go","file.line":702},"message":"Home path: [/usr/share/apm-server] Config path: [/usr/share/apm-server] Data path: [/usr/share/apm-server/data] Logs path: [/usr/share/apm-server/logs]","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-09-02T00:30:10.843Z","log.origin":{"file.name":"instance/beat.go","file.line":710},"message":"Beat ID: e1cd9361-c2cd-4cb2-82c0-8a2906ad5eec","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-09-02T00:30:10.843Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1076},"message":"Beat info","service.name":"apm-server","system_info":{"beat":{"path":{"config":"/usr/share/apm-server","data":"/usr/share/apm-server/data","home":"/usr/share/apm-server","logs":"/usr/share/apm-server/logs"},"type":"apm-server","uuid":"e1cd9361-c2cd-4cb2-82c0-8a2906ad5eec"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-09-02T00:30:10.843Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1085},"message":"Build info","service.name":"apm-server","system_info":{"build":{"commit":"54b929734617c29cf0b5415d4f1e31e5c080b7ad","libbeat":"8.4.0","time":"2022-08-25T18:49:04.000Z","version":"8.4.1"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-09-02T00:30:10.843Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1088},"message":"Go runtime info","service.name":"apm-server","system_info":{"go":{"os":"linux","arch":"amd64","max_procs":4,"version":"go1.17.12"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-09-02T00:30:10.844Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1092},"message":"Host info","service.name":"apm-server","system_info":{"host":{"architecture":"x86_64","boot_time":"2022-08-30T15:09:12Z","containerized":true,"name":"apm-apm-server-8b87fd85-tscs6","ip":["127.0.0.1/8","::1/128","10.12.249.167/21","fe80::5424:72ff:fed0:856e/64"],"kernel_version":"5.4.0-1089-azure","mac":["56:24:72:d0:85:6e"],"os":{"type":"linux","family":"debian","platform":"ubuntu","name":"Ubuntu","version":"20.04.4 LTS (Focal Fossa)","major":20,"minor":4,"patch":4,"codename":"focal"},"timezone":"UTC","timezone_offset_sec":0},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-09-02T00:30:10.844Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1121},"message":"Process info","service.name":"apm-server","system_info":{"process":{"capabilities":{"inheritable":null,"permitted":null,"effective":null,"bounding":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"ambient":null},"cwd":"/usr/share/apm-server","exe":"/usr/share/apm-server/apm-server","name":"apm-server","pid":8,"ppid":1,"seccomp":{"mode":"disabled","no_new_privs":false},"start_time":"2022-09-02T00:30:10.220Z"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-09-02T00:30:10.844Z","log.origin":{"file.name":"instance/beat.go","file.line":293},"message":"Setup Beat: apm-server; Version: 8.4.1","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-09-02T00:30:10.847Z","log.logger":"esclientleg","log.origin":{"file.name":"eslegclient/connection.go","file.line":108},"message":"elasticsearch url: https://elastic.<removed>.com:9200","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-09-02T00:30:10.847Z","log.logger":"publisher","log.origin":{"file.name":"pipeline/module.go","file.line":113},"message":"Beat name: apm-apm-server-8b87fd85-tscs6","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-09-02T00:30:10.848Z","log.origin":{"file.name":"instance/beat.go","file.line":470},"message":"apm-server start running.","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-09-02T00:30:10.848Z","log.logger":"beater","log.origin":{"file.name":"beater/http.go","file.line":144},"message":"Listening on: [::]:8200","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-09-02T00:30:10.848Z","log.logger":"beater","log.origin":{"file.name":"beater/beater.go","file.line":1073},"message":"maxprocs: Updating GOMAXPROCS=1: determined from CPU quota","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-09-02T00:30:10.850Z","log.logger":"beater","log.origin":{"file.name":"beater/waitready.go","file.line":40},"message":"blocking ingestion until all preconditions are satisfied","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-09-02T00:30:10.850Z","log.logger":"beater","log.origin":{"file.name":"apm-server/main.go","file.line":64},"message":"creating transaction metrics aggregation with config: {Interval:1m0s MaxTransactionGroups:10000 HDRHistogramSignificantFigures:2}","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-09-02T00:30:10.853Z","log.logger":"beater","log.origin":{"file.name":"apm-server/main.go","file.line":79},"message":"creating service destinations aggregation with config: {Interval:1m0s MaxGroups:10000}","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-09-02T00:30:10.854Z","log.logger":"handler","log.origin":{"file.name":"api/mux.go","file.line":140},"message":"Path / added to request handler","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-09-02T00:30:10.854Z","log.logger":"handler","log.origin":{"file.name":"api/mux.go","file.line":140},"message":"Path /config/v1/agents added to request handler","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-09-02T00:30:10.854Z","log.logger":"handler","log.origin":{"file.name":"api/mux.go","file.line":140},"message":"Path /config/v1/rum/agents added to request handler","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-09-02T00:30:10.854Z","log.logger":"handler","log.origin":{"file.name":"api/mux.go","file.line":140},"message":"Path /intake/v2/rum/events added to request handler","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-09-02T00:30:10.854Z","log.logger":"handler","log.origin":{"file.name":"api/mux.go","file.line":140},"message":"Path /intake/v3/rum/events added to request handler","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-09-02T00:30:10.854Z","log.logger":"handler","log.origin":{"file.name":"api/mux.go","file.line":140},"message":"Path /intake/v2/events added to request handler","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-09-02T00:30:10.854Z","log.logger":"handler","log.origin":{"file.name":"api/mux.go","file.line":140},"message":"Path /intake/v2/profile added to request handler","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-09-02T00:30:10.854Z","log.logger":"handler","log.origin":{"file.name":"api/mux.go","file.line":140},"message":"Path /v1/traces added to request handler","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-09-02T00:30:10.854Z","log.logger":"handler","log.origin":{"file.name":"api/mux.go","file.line":140},"message":"Path /v1/metrics added to request handler","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-09-02T00:30:10.854Z","log.logger":"handler","log.origin":{"file.name":"api/mux.go","file.line":140},"message":"Path /v1/logs added to request handler","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-09-02T00:30:10.854Z","log.logger":"beater","log.origin":{"file.name":"beater/server.go","file.line":233},"message":"Starting apm-server [54b929734617c29cf0b5415d4f1e31e5c080b7ad built 2022-08-25 18:49:04 +0000 UTC]. Hit CTRL-C to stop it.","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-09-02T00:30:10.854Z","log.logger":"beater","log.origin":{"file.name":"beater/http.go","file.line":96},"message":"RUM endpoints disabled.","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-09-02T00:30:10.854Z","log.logger":"beater","log.origin":{"file.name":"beater/http.go","file.line":106},"message":"SSL disabled.","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-09-02T00:30:46.476Z","log.logger":"request","log.origin":{"file.name":"middleware/log_middleware.go","file.line":61},"message":"request ok","service.name":"apm-server","url.original":"/","http.request.method":"GET","user_agent.original":"kube-probe/1.23","source.address":"10.12.249.148","http.request.body.bytes":0,"http.request.id":"7e3783a4-fb1f-4015-8fcc-dfb825d43a7a","event.duration":102870,"http.response.status_code":200,"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-09-02T00:30:46.477Z","log.logger":"request","log.origin":{"file.name":"middleware/log_middleware.go","file.line":61},"message":"request ok","service.name":"apm-server","url.original":"/","http.request.method":"GET","user_agent.original":"kube-probe/1.23","source.address":"10.12.249.148","http.request.body.bytes":0,"http.request.id":"99203647-0b23-4bb4-bc8b-c85dcbe79fbb","event.duration":46645,"http.response.status_code":200,"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-09-02T00:30:56.476Z","log.logger":"request","log.origin":{"file.name":"middleware/log_middleware.go","file.line":61},"message":"request ok","service.name":"apm-server","url.original":"/","http.request.method":"GET","user_agent.original":"kube-probe/1.23","source.address":"10.12.249.148","http.request.body.bytes":0,"http.request.id":"f4cefe04-9ca3-4662-ae95-b1835fae63bf","event.duration":105186,"http.response.status_code":200,"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-09-02T00:30:56.477Z","log.logger":"request","log.origin":{"file.name":"middleware/log_middleware.go","file.line":61},"message":"request ok","service.name":"apm-server","url.original":"/","http.request.method":"GET","user_agent.original":"kube-probe/1.23","source.address":"10.12.249.148","http.request.body.bytes":0,"http.request.id":"2d7db3f1-1c46-4302-b0bb-9c958b743914","event.duration":64670,"http.response.status_code":200,"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-09-02T00:30:56.871Z","log.logger":"request","log.origin":{"file.name":"middleware/log_middleware.go","file.line":61},"message":"handled request","service.name":"apm-server","url.original":"/v1/traces","http.request.method":"POST","user_agent.original":"OpenTelemetry Collector Contrib/0.56.0 (linux/amd64)","source.address":"10.12.248.56","http.request.body.bytes":813,"http.request.id":"c1e583d0-fcef-45f1-b24e-54870fa856da","event.duration":146745,"http.response.status_code":200,"ecs.version":"1.6.0"}
**{"log.level":"info","@timestamp":"2022-09-02T00:31:03.687Z","log.logger":"request","log.origin":{"file.name":"middleware/log_middleware.go","file.line":61},"message":"handled request","service.name":"apm-server","url.original":"/v1/traces","http.request.method":"POST","user_agent.original":"OpenTelemetry Collector Contrib/0.56.0 (linux/amd64)","source.address":"10.12.248.56","http.request.body.bytes":815,"http.request.id":"ab25dfa0-c6bb-4e8f-ba75-f665678ae221","event.duration":64960,"http.response.status_code":200,"ecs.version":"1.6.0"}**
{"log.level":"info","@timestamp":"2022-09-02T00:31:06.477Z","log.logger":"request","log.origin":{"file.name":"middleware/log_middleware.go","file.line":61},"message":"request ok","service.name":"apm-server","url.original":"/","http.request.method":"GET","user_agent.original":"kube-probe/1.23","source.address":"10.12.249.148","http.request.body.bytes":0,"http.request.id":"2839fceb-1534-483e-96bb-6d7f349fe7a9","event.duration":104900,"http.response.status_code":200,"ecs.version":"1.6.0"}

collector logs:

2022/09/02 00:38:49 proto: duplicate proto type registered: jaeger.api_v2.PostSpansRequest
2022/09/02 00:38:49 proto: duplicate proto type registered: jaeger.api_v2.PostSpansResponse
2022-09-02T00:38:49.125Z        info    service/telemetry.go:102        Setting up own telemetry...
2022-09-02T00:38:49.126Z        info    service/telemetry.go:137        Serving Prometheus metrics      {"address": "0.0.0.0:8888", "level": "basic"}
2022-09-02T00:38:49.126Z        info    components/components.go:30     In development component. May change in the future.     {"kind": "exporter", "data_type": "traces", "name": "logging", "stability": "in development"}
2022-09-02T00:38:49.126Z        info    components/components.go:30     Deprecated component. Will be removed in future releases.       {"kind": "exporter", "data_type": "traces", "name": "elastic", "stability": "deprecated"}
2022-09-02T00:38:49.126Z        warn    elasticexporter@v0.56.0/factory.go:53   elastic exporter is deprecated and will be removed in future versions.  {"kind": "exporter", "data_type": "traces", "name": "elastic"}
2022-09-02T00:38:49.126Z        info    extensions/extensions.go:42     Starting extensions...
2022-09-02T00:38:49.126Z        info    extensions/extensions.go:45     Extension is starting...        {"kind": "extension", "name": "memory_ballast"}
2022-09-02T00:38:49.127Z        info    ballastextension/memory_ballast.go:52   Setting memory ballast  {"kind": "extension", "name": "memory_ballast", "MiBs": 0}
2022-09-02T00:38:49.127Z        info    extensions/extensions.go:49     Extension started.      {"kind": "extension", "name": "memory_ballast"}
2022-09-02T00:38:49.127Z        info    extensions/extensions.go:45     Extension is starting...        {"kind": "extension", "name": "health_check"}
2022-09-02T00:38:49.127Z        info    healthcheckextension@v0.56.0/healthcheckextension.go:44 Starting health_check extension {"kind": "extension", "name": "health_check", "config": {"Port":0,"TCPAddr":{"Endpoint":"0.0.0.0:13133"},"Path":"/","CheckCollectorPipeline":{"Enabled":false,"Interval":"5m","ExporterFailureThreshold":5}}}
2022-09-02T00:38:49.127Z        info    extensions/extensions.go:49     Extension started.      {"kind": "extension", "name": "health_check"}
2022-09-02T00:38:49.127Z        info    pipelines/pipelines.go:74       Starting exporters...
2022-09-02T00:38:49.127Z        info    pipelines/pipelines.go:78       Exporter is starting... {"kind": "exporter", "data_type": "traces", "name": "logging"}
2022-09-02T00:38:49.127Z        info    pipelines/pipelines.go:82       Exporter started.       {"kind": "exporter", "data_type": "traces", "name": "logging"}
2022-09-02T00:38:49.127Z        info    pipelines/pipelines.go:78       Exporter is starting... {"kind": "exporter", "data_type": "traces", "name": "elastic"}
2022-09-02T00:38:49.127Z        info    pipelines/pipelines.go:82       Exporter started.       {"kind": "exporter", "data_type": "traces", "name": "elastic"}
2022-09-02T00:38:49.127Z        info    pipelines/pipelines.go:78       Exporter is starting... {"kind": "exporter", "data_type": "traces", "name": "otlphttp/insecure_no_verify"}
2022-09-02T00:38:49.127Z        info    pipelines/pipelines.go:82       Exporter started.       {"kind": "exporter", "data_type": "traces", "name": "otlphttp/insecure_no_verify"}
2022-09-02T00:38:49.127Z        info    pipelines/pipelines.go:86       Starting processors...
2022-09-02T00:38:49.127Z        info    pipelines/pipelines.go:90       Processor is starting...        {"kind": "processor", "name": "attributes", "pipeline": "traces"}
2022-09-02T00:38:49.127Z        info    pipelines/pipelines.go:94       Processor started.      {"kind": "processor", "name": "attributes", "pipeline": "traces"}
2022-09-02T00:38:49.127Z        info    pipelines/pipelines.go:98       Starting receivers...
2022-09-02T00:38:49.127Z        info    pipelines/pipelines.go:102      Receiver is starting... {"kind": "receiver", "name": "otlp", "pipeline": "traces"}
2022-09-02T00:38:49.127Z        info    otlpreceiver/otlp.go:70 Starting GRPC server on endpoint 0.0.0.0:4317   {"kind": "receiver", "name": "otlp", "pipeline": "traces"}
2022-09-02T00:38:49.127Z        info    otlpreceiver/otlp.go:88 Starting HTTP server on endpoint 0.0.0.0:4318   {"kind": "receiver", "name": "otlp", "pipeline": "traces"}
2022-09-02T00:38:49.127Z        info    pipelines/pipelines.go:106      Receiver started.       {"kind": "receiver", "name": "otlp", "pipeline": "traces"}
2022-09-02T00:38:49.127Z        info    healthcheck/handler.go:129      Health Check state change       {"kind": "extension", "name": "health_check", "status": "ready"}
2022-09-02T00:38:49.127Z        info    service/collector.go:215        Starting otelcol-contrib...     {"Version": "0.56.0", "NumCPU": 4}
2022-09-02T00:38:49.127Z        info    service/collector.go:128        Everything is ready. Begin running and processing data.
2022-09-02T00:41:35.755Z        info    TracesExporter  {"kind": "exporter", "data_type": "traces", "name": "logging", "#spans": 5}
2022-09-02T00:41:35.755Z        info    ResourceSpans #0
Resource SchemaURL:
Resource labels:
     -> service.name: STRING(PWDI Server)
     -> service.instance.id: STRING(6bc70ae6-f0c0-44da-9ef6-07eef3f2b2f8)
ScopeSpans #0
ScopeSpans SchemaURL:
InstrumentationScope PWDI
Span #0
    Trace ID       : 7b12a75510ff3c4ebc59130bcefa7615
    Parent ID      :
    ID             : 14ad637bb8b33a40
    Name           : PwDbOperation
    Kind           : SPAN_KIND_SERVER
    Start time     : 2022-09-02 00:41:31.6457119 +0000 UTC
    End time       : 2022-09-02 00:41:31.649067 +0000 UTC
    Status code    : STATUS_CODE_UNSET
    Status message :
Attributes:
     -> session: STRING(0xb1ea66e0)
     -> dbAction: STRING(ExecuteSelect)
     -> sqlCmd: STRING(select schema_name())
     -> datasource: STRING(pwdsn_db0)
     -> bmap.plan_id: STRING(63075731126d24faff9b56ca)
     -> bmap.plan_name: STRING(hj-otel3)
Span #1
    Trace ID       : c474cb5be6aad542829d6afc5f8cd85f
    Parent ID      :
    ID             : de61c58f5cf0c345
    Name           : PwDbOperation
    Kind           : SPAN_KIND_SERVER
    Start time     : 2022-09-02 00:41:31.7186582 +0000 UTC
    End time       : 2022-09-02 00:41:31.7221124 +0000 UTC
    Status code    : STATUS_CODE_UNSET
    Status message :
Attributes:
     -> session: STRING(0xb1ea66e0)
     -> dbAction: STRING(ExecuteSelect)
     -> sqlCmd: STRING(select count(*) where ('A' <> 'a'))
     -> datasource: STRING(pwdsn_db0)
     -> bmap.plan_id: STRING(63075731126d24faff9b56ca)
     -> bmap.plan_name: STRING(hj-otel3)
Span #2
    Trace ID       : 6b23cbb675479e4b84bb6567d2836115
    Parent ID      :
    ID             : 5d91048f7506414a
    Name           : PwDbOperation
    Kind           : SPAN_KIND_SERVER
    Start time     : 2022-09-02 00:41:31.7225876 +0000 UTC
    End time       : 2022-09-02 00:41:31.7255151 +0000 UTC
    Status code    : STATUS_CODE_UNSET
    Status message :
Attributes:
     -> session: STRING(0xb1ea66e0)
     -> dbAction: STRING(ExecuteSelect)
     -> sqlCmd: STRING(select cast(SERVERPROPERTY('EngineEdition') as int))
     -> datasource: STRING(pwdsn_db0)
     -> bmap.plan_id: STRING(63075731126d24faff9b56ca)
     -> bmap.plan_name: STRING(hj-otel3)
Span #3
    Trace ID       : ffc7de2ff1eeec49884d7d9803597217
    Parent ID      :
    ID             : d227094270ed3244
    Name           : PwDbOperation
    Kind           : SPAN_KIND_SERVER
    Start time     : 2022-09-02 00:41:31.7387606 +0000 UTC
    End time       : 2022-09-02 00:41:31.7417716 +0000 UTC
    Status code    : STATUS_CODE_UNSET
    Status message :
Attributes:
     -> session: STRING(0xb1ea66e0)
     -> dbAction: STRING(ExecuteSelect)
     -> sqlCmd: STRING(select o_verno,o_rectype,o_release,o_major,o_minor,o_build,o_updatetime,o_text from dms_ver  where o_rectype = 10 order by o_verno)
     -> datasource: STRING(pwdsn_db0)
     -> bmap.plan_id: STRING(63075731126d24faff9b56ca)
     -> bmap.plan_name: STRING(hj-otel3)
Span #4
    Trace ID       : 7033875c3332ed4698f08c6691073f95
    Parent ID      :
    ID             : 6a6a0e1ca26b8e48
    Name           : PwDbOperation
    Kind           : SPAN_KIND_SERVER
    Start time     : 2022-09-02 00:41:31.8091887 +0000 UTC
    End time       : 2022-09-02 00:41:31.8125511 +0000 UTC
    Status code    : STATUS_CODE_UNSET
    Status message :
Attributes:
     -> session: STRING(0xb1ea66e0)
     -> dbAction: STRING(ExecuteSelect)
     -> sqlCmd: STRING(select INDEXDOCINFOID,IMPORT from EC_INDEXDOCINFO where (DOC_GUID not in (select o_docguid from dms_doc)) order by IMPORT)
     -> datasource: STRING(pwdsn_db0)
     -> bmap.plan_id: STRING(63075731126d24faff9b56ca)
     -> bmap.plan_name: STRING(hj-otel3)
        {"kind": "exporter", "data_type": "traces", "name": "logging"}
2022-09-02T00:41:35.812Z        error   exporterhelper/queued_retry.go:183      **Exporting failed. The error is not retryable. Dropping data.    {"kind": "exporter", "data_type": "traces", "name": "otlphttp/insecure_no_verify", "error": "Permanent error: error exporting items, request to https://apm.<removed>:8200/v1/traces responded with HTTP Status Code 400, Message=proto: illegal wireType 7, Details=[]", "dropped_items": 5**}
go.opentelemetry.io/collector/exporter/exporterhelper.(*retrySender).send
        go.opentelemetry.io/collector@v0.56.0/exporter/exporterhelper/queued_retry.go:183
go.opentelemetry.io/collector/exporter/exporterhelper.(*tracesExporterWithObservability).send
        go.opentelemetry.io/collector@v0.56.0/exporter/exporterhelper/traces.go:134
go.opentelemetry.io/collector/exporter/exporterhelper.(*queuedRetrySender).start.func1
        go.opentelemetry.io/collector@v0.56.0/exporter/exporterhelper/queued_retry_inmemory.go:119
go.opentelemetry.io/collector/exporter/exporterhelper/internal.consumerFunc.consume
        go.opentelemetry.io/collector@v0.56.0/exporter/exporterhelper/internal/bounded_memory_queue.go:82
go.opentelemetry.io/collector/exporter/exporterhelper/internal.(*boundedMemoryQueue).StartConsumers.func2
        go.opentelemetry.io/collector@v0.56.0/exporter/exporterhelper/internal/bounded_memory_queue.go:69
2022-09-02T00:43:25.011Z        info    TracesExporter  {"kind": "exporter", "data_type": "traces", "name": "logging", "#spans": 5}
2022-09-02T00:43:25.011Z        info    ResourceSpans #0

continuing to above post:

if i use otlp exporter instead of otlphttp in collector config,

otlp/insecure:
        endpoint: https://apm.<removed>:8200
        tls:
          insecure: true

then I get HTTP 404

2022-09-02T08:48:48.433Z        error   exporterhelper/queued_retry.go:183      Exporting failed. The error is not retryable. Dropping data.    {"kind": "exporter", "data_type": "traces", "name": "otlp/insecure", "error": "Permanent error: rpc error: code = Unimplemented desc = unexpected HTTP status code received from server: 404 (Not Found); transport: received unexpected content-type \"text/plain; charset=utf-8\"", "dropped_items": 5}

if I use above exporter config without https:// prefix as given in elastic-apm docs,

otlp/insecure:
        endpoint: apm.<removed>:8200
        tls:
          insecure: true

then I get below error:

{"kind": "exporter", "data_type": "traces", "name": "logging"}
2022-09-02T08:53:51.584Z        info    exporterhelper/queued_retry.go:215      Exporting failed. Will retry the request after interval.        {"kind": "exporter", "data_type": "traces", "name": "otlp/insecure", "error": "rpc error: code = Unavailable desc = connection closed before server preface received", "interval": "4.055431468s"}

It appears that we're not handling the otel-collector's default compression properly: OTLP/HTTP requests fail on compressed payloads · Issue #9036 · elastic/apm-server · GitHub

Until that is fixed, you can work around this by adding compression: none in the otlphttp config block.

1 Like

Thanks a lot @axw this fixed my issue :slightly_smiling_face:

1 Like