Hi,
We have our application instrumented to push traces on otlp to open telemetry collector and we are then exporting traces to elastic APM. we were earlier using "elastic" exporter to push traces to elastic APM server directly and it was working perfectly fine until it was deprecated and we are forced to look for alternatives.
as per latest documentation, we are now trying to use otlp or otlphttp exporter in our opentelemetry collector, however it is not working. latest status is we are getting HTTP 400 while using otlphttp exporter.
I am using ingress with load balancer service type for APM server for SSL termination. opentelemetry collector is using clusterip service type.
Stack :
Open telemetry collector image: docker.io/otel/opentelemetry-collector-contrib:latest
elastic APM Server: 8.4.1
otel-collector config:
exporters:
logging:
logLevel: debug
otlphttp/insecure_no_verify:
endpoint: https://apm.<removed>:8200
tls:
insecure: false
insecure_skip_verify: true
extensions:
health_check: {}
memory_ballast: {}
processors:
attributes:
actions:
- action: insert
key: bmap.plan_id
value: 63075731126d24faff9b56ca
- action: insert
key: bmap.plan_name
value: hj-otel3
batch: {}
memory_limiter:
check_interval: 5s
limit_mib: 1638
spike_limit_mib: 512
receivers:
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
http:
endpoint: 0.0.0.0:4318
service:
extensions:
- health_check
- memory_ballast
pipelines:
traces:
exporters:
- logging
- elastic
- otlphttp/insecure_no_verify
processors:
- attributes
receivers:
- otlp
telemetry:
metrics:
address: 0.0.0.0:8888
APM Server config:
apm-server:
host: "0.0.0.0:8200"
ssl:
enabled: true
certificate: "/usr/share/apm-server/certs/tls.crt"
key: "/usr/share/apm-server/certs/tls.key"
supported_protocols: [TLSv1.1, TLSv1.2, TLSv1.3]
client_authentication: "none"
queue: {}
auth:
anonymous:
enabled: true
output.elasticsearch:
hosts: ["https://elastic.<removed>:9200"]
username: "elastic"
password: "<removed>"
APM Server logs:
{"log.level":"info","@timestamp":"2022-09-02T00:30:10.832Z","log.origin":{"file.name":"instance/beat.go","file.line":702},"message":"Home path: [/usr/share/apm-server] Config path: [/usr/share/apm-server] Data path: [/usr/share/apm-server/data] Logs path: [/usr/share/apm-server/logs]","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-09-02T00:30:10.843Z","log.origin":{"file.name":"instance/beat.go","file.line":710},"message":"Beat ID: e1cd9361-c2cd-4cb2-82c0-8a2906ad5eec","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-09-02T00:30:10.843Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1076},"message":"Beat info","service.name":"apm-server","system_info":{"beat":{"path":{"config":"/usr/share/apm-server","data":"/usr/share/apm-server/data","home":"/usr/share/apm-server","logs":"/usr/share/apm-server/logs"},"type":"apm-server","uuid":"e1cd9361-c2cd-4cb2-82c0-8a2906ad5eec"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-09-02T00:30:10.843Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1085},"message":"Build info","service.name":"apm-server","system_info":{"build":{"commit":"54b929734617c29cf0b5415d4f1e31e5c080b7ad","libbeat":"8.4.0","time":"2022-08-25T18:49:04.000Z","version":"8.4.1"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-09-02T00:30:10.843Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1088},"message":"Go runtime info","service.name":"apm-server","system_info":{"go":{"os":"linux","arch":"amd64","max_procs":4,"version":"go1.17.12"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-09-02T00:30:10.844Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1092},"message":"Host info","service.name":"apm-server","system_info":{"host":{"architecture":"x86_64","boot_time":"2022-08-30T15:09:12Z","containerized":true,"name":"apm-apm-server-8b87fd85-tscs6","ip":["127.0.0.1/8","::1/128","10.12.249.167/21","fe80::5424:72ff:fed0:856e/64"],"kernel_version":"5.4.0-1089-azure","mac":["56:24:72:d0:85:6e"],"os":{"type":"linux","family":"debian","platform":"ubuntu","name":"Ubuntu","version":"20.04.4 LTS (Focal Fossa)","major":20,"minor":4,"patch":4,"codename":"focal"},"timezone":"UTC","timezone_offset_sec":0},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-09-02T00:30:10.844Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1121},"message":"Process info","service.name":"apm-server","system_info":{"process":{"capabilities":{"inheritable":null,"permitted":null,"effective":null,"bounding":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"ambient":null},"cwd":"/usr/share/apm-server","exe":"/usr/share/apm-server/apm-server","name":"apm-server","pid":8,"ppid":1,"seccomp":{"mode":"disabled","no_new_privs":false},"start_time":"2022-09-02T00:30:10.220Z"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-09-02T00:30:10.844Z","log.origin":{"file.name":"instance/beat.go","file.line":293},"message":"Setup Beat: apm-server; Version: 8.4.1","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-09-02T00:30:10.847Z","log.logger":"esclientleg","log.origin":{"file.name":"eslegclient/connection.go","file.line":108},"message":"elasticsearch url: https://elastic.<removed>.com:9200","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-09-02T00:30:10.847Z","log.logger":"publisher","log.origin":{"file.name":"pipeline/module.go","file.line":113},"message":"Beat name: apm-apm-server-8b87fd85-tscs6","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-09-02T00:30:10.848Z","log.origin":{"file.name":"instance/beat.go","file.line":470},"message":"apm-server start running.","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-09-02T00:30:10.848Z","log.logger":"beater","log.origin":{"file.name":"beater/http.go","file.line":144},"message":"Listening on: [::]:8200","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-09-02T00:30:10.848Z","log.logger":"beater","log.origin":{"file.name":"beater/beater.go","file.line":1073},"message":"maxprocs: Updating GOMAXPROCS=1: determined from CPU quota","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-09-02T00:30:10.850Z","log.logger":"beater","log.origin":{"file.name":"beater/waitready.go","file.line":40},"message":"blocking ingestion until all preconditions are satisfied","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-09-02T00:30:10.850Z","log.logger":"beater","log.origin":{"file.name":"apm-server/main.go","file.line":64},"message":"creating transaction metrics aggregation with config: {Interval:1m0s MaxTransactionGroups:10000 HDRHistogramSignificantFigures:2}","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-09-02T00:30:10.853Z","log.logger":"beater","log.origin":{"file.name":"apm-server/main.go","file.line":79},"message":"creating service destinations aggregation with config: {Interval:1m0s MaxGroups:10000}","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-09-02T00:30:10.854Z","log.logger":"handler","log.origin":{"file.name":"api/mux.go","file.line":140},"message":"Path / added to request handler","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-09-02T00:30:10.854Z","log.logger":"handler","log.origin":{"file.name":"api/mux.go","file.line":140},"message":"Path /config/v1/agents added to request handler","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-09-02T00:30:10.854Z","log.logger":"handler","log.origin":{"file.name":"api/mux.go","file.line":140},"message":"Path /config/v1/rum/agents added to request handler","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-09-02T00:30:10.854Z","log.logger":"handler","log.origin":{"file.name":"api/mux.go","file.line":140},"message":"Path /intake/v2/rum/events added to request handler","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-09-02T00:30:10.854Z","log.logger":"handler","log.origin":{"file.name":"api/mux.go","file.line":140},"message":"Path /intake/v3/rum/events added to request handler","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-09-02T00:30:10.854Z","log.logger":"handler","log.origin":{"file.name":"api/mux.go","file.line":140},"message":"Path /intake/v2/events added to request handler","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-09-02T00:30:10.854Z","log.logger":"handler","log.origin":{"file.name":"api/mux.go","file.line":140},"message":"Path /intake/v2/profile added to request handler","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-09-02T00:30:10.854Z","log.logger":"handler","log.origin":{"file.name":"api/mux.go","file.line":140},"message":"Path /v1/traces added to request handler","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-09-02T00:30:10.854Z","log.logger":"handler","log.origin":{"file.name":"api/mux.go","file.line":140},"message":"Path /v1/metrics added to request handler","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-09-02T00:30:10.854Z","log.logger":"handler","log.origin":{"file.name":"api/mux.go","file.line":140},"message":"Path /v1/logs added to request handler","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-09-02T00:30:10.854Z","log.logger":"beater","log.origin":{"file.name":"beater/server.go","file.line":233},"message":"Starting apm-server [54b929734617c29cf0b5415d4f1e31e5c080b7ad built 2022-08-25 18:49:04 +0000 UTC]. Hit CTRL-C to stop it.","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-09-02T00:30:10.854Z","log.logger":"beater","log.origin":{"file.name":"beater/http.go","file.line":96},"message":"RUM endpoints disabled.","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-09-02T00:30:10.854Z","log.logger":"beater","log.origin":{"file.name":"beater/http.go","file.line":106},"message":"SSL disabled.","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-09-02T00:30:46.476Z","log.logger":"request","log.origin":{"file.name":"middleware/log_middleware.go","file.line":61},"message":"request ok","service.name":"apm-server","url.original":"/","http.request.method":"GET","user_agent.original":"kube-probe/1.23","source.address":"10.12.249.148","http.request.body.bytes":0,"http.request.id":"7e3783a4-fb1f-4015-8fcc-dfb825d43a7a","event.duration":102870,"http.response.status_code":200,"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-09-02T00:30:46.477Z","log.logger":"request","log.origin":{"file.name":"middleware/log_middleware.go","file.line":61},"message":"request ok","service.name":"apm-server","url.original":"/","http.request.method":"GET","user_agent.original":"kube-probe/1.23","source.address":"10.12.249.148","http.request.body.bytes":0,"http.request.id":"99203647-0b23-4bb4-bc8b-c85dcbe79fbb","event.duration":46645,"http.response.status_code":200,"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-09-02T00:30:56.476Z","log.logger":"request","log.origin":{"file.name":"middleware/log_middleware.go","file.line":61},"message":"request ok","service.name":"apm-server","url.original":"/","http.request.method":"GET","user_agent.original":"kube-probe/1.23","source.address":"10.12.249.148","http.request.body.bytes":0,"http.request.id":"f4cefe04-9ca3-4662-ae95-b1835fae63bf","event.duration":105186,"http.response.status_code":200,"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-09-02T00:30:56.477Z","log.logger":"request","log.origin":{"file.name":"middleware/log_middleware.go","file.line":61},"message":"request ok","service.name":"apm-server","url.original":"/","http.request.method":"GET","user_agent.original":"kube-probe/1.23","source.address":"10.12.249.148","http.request.body.bytes":0,"http.request.id":"2d7db3f1-1c46-4302-b0bb-9c958b743914","event.duration":64670,"http.response.status_code":200,"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-09-02T00:30:56.871Z","log.logger":"request","log.origin":{"file.name":"middleware/log_middleware.go","file.line":61},"message":"handled request","service.name":"apm-server","url.original":"/v1/traces","http.request.method":"POST","user_agent.original":"OpenTelemetry Collector Contrib/0.56.0 (linux/amd64)","source.address":"10.12.248.56","http.request.body.bytes":813,"http.request.id":"c1e583d0-fcef-45f1-b24e-54870fa856da","event.duration":146745,"http.response.status_code":200,"ecs.version":"1.6.0"}
**{"log.level":"info","@timestamp":"2022-09-02T00:31:03.687Z","log.logger":"request","log.origin":{"file.name":"middleware/log_middleware.go","file.line":61},"message":"handled request","service.name":"apm-server","url.original":"/v1/traces","http.request.method":"POST","user_agent.original":"OpenTelemetry Collector Contrib/0.56.0 (linux/amd64)","source.address":"10.12.248.56","http.request.body.bytes":815,"http.request.id":"ab25dfa0-c6bb-4e8f-ba75-f665678ae221","event.duration":64960,"http.response.status_code":200,"ecs.version":"1.6.0"}**
{"log.level":"info","@timestamp":"2022-09-02T00:31:06.477Z","log.logger":"request","log.origin":{"file.name":"middleware/log_middleware.go","file.line":61},"message":"request ok","service.name":"apm-server","url.original":"/","http.request.method":"GET","user_agent.original":"kube-probe/1.23","source.address":"10.12.249.148","http.request.body.bytes":0,"http.request.id":"2839fceb-1534-483e-96bb-6d7f349fe7a9","event.duration":104900,"http.response.status_code":200,"ecs.version":"1.6.0"}
collector logs:
2022/09/02 00:38:49 proto: duplicate proto type registered: jaeger.api_v2.PostSpansRequest
2022/09/02 00:38:49 proto: duplicate proto type registered: jaeger.api_v2.PostSpansResponse
2022-09-02T00:38:49.125Z info service/telemetry.go:102 Setting up own telemetry...
2022-09-02T00:38:49.126Z info service/telemetry.go:137 Serving Prometheus metrics {"address": "0.0.0.0:8888", "level": "basic"}
2022-09-02T00:38:49.126Z info components/components.go:30 In development component. May change in the future. {"kind": "exporter", "data_type": "traces", "name": "logging", "stability": "in development"}
2022-09-02T00:38:49.126Z info components/components.go:30 Deprecated component. Will be removed in future releases. {"kind": "exporter", "data_type": "traces", "name": "elastic", "stability": "deprecated"}
2022-09-02T00:38:49.126Z warn elasticexporter@v0.56.0/factory.go:53 elastic exporter is deprecated and will be removed in future versions. {"kind": "exporter", "data_type": "traces", "name": "elastic"}
2022-09-02T00:38:49.126Z info extensions/extensions.go:42 Starting extensions...
2022-09-02T00:38:49.126Z info extensions/extensions.go:45 Extension is starting... {"kind": "extension", "name": "memory_ballast"}
2022-09-02T00:38:49.127Z info ballastextension/memory_ballast.go:52 Setting memory ballast {"kind": "extension", "name": "memory_ballast", "MiBs": 0}
2022-09-02T00:38:49.127Z info extensions/extensions.go:49 Extension started. {"kind": "extension", "name": "memory_ballast"}
2022-09-02T00:38:49.127Z info extensions/extensions.go:45 Extension is starting... {"kind": "extension", "name": "health_check"}
2022-09-02T00:38:49.127Z info healthcheckextension@v0.56.0/healthcheckextension.go:44 Starting health_check extension {"kind": "extension", "name": "health_check", "config": {"Port":0,"TCPAddr":{"Endpoint":"0.0.0.0:13133"},"Path":"/","CheckCollectorPipeline":{"Enabled":false,"Interval":"5m","ExporterFailureThreshold":5}}}
2022-09-02T00:38:49.127Z info extensions/extensions.go:49 Extension started. {"kind": "extension", "name": "health_check"}
2022-09-02T00:38:49.127Z info pipelines/pipelines.go:74 Starting exporters...
2022-09-02T00:38:49.127Z info pipelines/pipelines.go:78 Exporter is starting... {"kind": "exporter", "data_type": "traces", "name": "logging"}
2022-09-02T00:38:49.127Z info pipelines/pipelines.go:82 Exporter started. {"kind": "exporter", "data_type": "traces", "name": "logging"}
2022-09-02T00:38:49.127Z info pipelines/pipelines.go:78 Exporter is starting... {"kind": "exporter", "data_type": "traces", "name": "elastic"}
2022-09-02T00:38:49.127Z info pipelines/pipelines.go:82 Exporter started. {"kind": "exporter", "data_type": "traces", "name": "elastic"}
2022-09-02T00:38:49.127Z info pipelines/pipelines.go:78 Exporter is starting... {"kind": "exporter", "data_type": "traces", "name": "otlphttp/insecure_no_verify"}
2022-09-02T00:38:49.127Z info pipelines/pipelines.go:82 Exporter started. {"kind": "exporter", "data_type": "traces", "name": "otlphttp/insecure_no_verify"}
2022-09-02T00:38:49.127Z info pipelines/pipelines.go:86 Starting processors...
2022-09-02T00:38:49.127Z info pipelines/pipelines.go:90 Processor is starting... {"kind": "processor", "name": "attributes", "pipeline": "traces"}
2022-09-02T00:38:49.127Z info pipelines/pipelines.go:94 Processor started. {"kind": "processor", "name": "attributes", "pipeline": "traces"}
2022-09-02T00:38:49.127Z info pipelines/pipelines.go:98 Starting receivers...
2022-09-02T00:38:49.127Z info pipelines/pipelines.go:102 Receiver is starting... {"kind": "receiver", "name": "otlp", "pipeline": "traces"}
2022-09-02T00:38:49.127Z info otlpreceiver/otlp.go:70 Starting GRPC server on endpoint 0.0.0.0:4317 {"kind": "receiver", "name": "otlp", "pipeline": "traces"}
2022-09-02T00:38:49.127Z info otlpreceiver/otlp.go:88 Starting HTTP server on endpoint 0.0.0.0:4318 {"kind": "receiver", "name": "otlp", "pipeline": "traces"}
2022-09-02T00:38:49.127Z info pipelines/pipelines.go:106 Receiver started. {"kind": "receiver", "name": "otlp", "pipeline": "traces"}
2022-09-02T00:38:49.127Z info healthcheck/handler.go:129 Health Check state change {"kind": "extension", "name": "health_check", "status": "ready"}
2022-09-02T00:38:49.127Z info service/collector.go:215 Starting otelcol-contrib... {"Version": "0.56.0", "NumCPU": 4}
2022-09-02T00:38:49.127Z info service/collector.go:128 Everything is ready. Begin running and processing data.
2022-09-02T00:41:35.755Z info TracesExporter {"kind": "exporter", "data_type": "traces", "name": "logging", "#spans": 5}
2022-09-02T00:41:35.755Z info ResourceSpans #0
Resource SchemaURL:
Resource labels:
-> service.name: STRING(PWDI Server)
-> service.instance.id: STRING(6bc70ae6-f0c0-44da-9ef6-07eef3f2b2f8)
ScopeSpans #0
ScopeSpans SchemaURL:
InstrumentationScope PWDI
Span #0
Trace ID : 7b12a75510ff3c4ebc59130bcefa7615
Parent ID :
ID : 14ad637bb8b33a40
Name : PwDbOperation
Kind : SPAN_KIND_SERVER
Start time : 2022-09-02 00:41:31.6457119 +0000 UTC
End time : 2022-09-02 00:41:31.649067 +0000 UTC
Status code : STATUS_CODE_UNSET
Status message :
Attributes:
-> session: STRING(0xb1ea66e0)
-> dbAction: STRING(ExecuteSelect)
-> sqlCmd: STRING(select schema_name())
-> datasource: STRING(pwdsn_db0)
-> bmap.plan_id: STRING(63075731126d24faff9b56ca)
-> bmap.plan_name: STRING(hj-otel3)
Span #1
Trace ID : c474cb5be6aad542829d6afc5f8cd85f
Parent ID :
ID : de61c58f5cf0c345
Name : PwDbOperation
Kind : SPAN_KIND_SERVER
Start time : 2022-09-02 00:41:31.7186582 +0000 UTC
End time : 2022-09-02 00:41:31.7221124 +0000 UTC
Status code : STATUS_CODE_UNSET
Status message :
Attributes:
-> session: STRING(0xb1ea66e0)
-> dbAction: STRING(ExecuteSelect)
-> sqlCmd: STRING(select count(*) where ('A' <> 'a'))
-> datasource: STRING(pwdsn_db0)
-> bmap.plan_id: STRING(63075731126d24faff9b56ca)
-> bmap.plan_name: STRING(hj-otel3)
Span #2
Trace ID : 6b23cbb675479e4b84bb6567d2836115
Parent ID :
ID : 5d91048f7506414a
Name : PwDbOperation
Kind : SPAN_KIND_SERVER
Start time : 2022-09-02 00:41:31.7225876 +0000 UTC
End time : 2022-09-02 00:41:31.7255151 +0000 UTC
Status code : STATUS_CODE_UNSET
Status message :
Attributes:
-> session: STRING(0xb1ea66e0)
-> dbAction: STRING(ExecuteSelect)
-> sqlCmd: STRING(select cast(SERVERPROPERTY('EngineEdition') as int))
-> datasource: STRING(pwdsn_db0)
-> bmap.plan_id: STRING(63075731126d24faff9b56ca)
-> bmap.plan_name: STRING(hj-otel3)
Span #3
Trace ID : ffc7de2ff1eeec49884d7d9803597217
Parent ID :
ID : d227094270ed3244
Name : PwDbOperation
Kind : SPAN_KIND_SERVER
Start time : 2022-09-02 00:41:31.7387606 +0000 UTC
End time : 2022-09-02 00:41:31.7417716 +0000 UTC
Status code : STATUS_CODE_UNSET
Status message :
Attributes:
-> session: STRING(0xb1ea66e0)
-> dbAction: STRING(ExecuteSelect)
-> sqlCmd: STRING(select o_verno,o_rectype,o_release,o_major,o_minor,o_build,o_updatetime,o_text from dms_ver where o_rectype = 10 order by o_verno)
-> datasource: STRING(pwdsn_db0)
-> bmap.plan_id: STRING(63075731126d24faff9b56ca)
-> bmap.plan_name: STRING(hj-otel3)
Span #4
Trace ID : 7033875c3332ed4698f08c6691073f95
Parent ID :
ID : 6a6a0e1ca26b8e48
Name : PwDbOperation
Kind : SPAN_KIND_SERVER
Start time : 2022-09-02 00:41:31.8091887 +0000 UTC
End time : 2022-09-02 00:41:31.8125511 +0000 UTC
Status code : STATUS_CODE_UNSET
Status message :
Attributes:
-> session: STRING(0xb1ea66e0)
-> dbAction: STRING(ExecuteSelect)
-> sqlCmd: STRING(select INDEXDOCINFOID,IMPORT from EC_INDEXDOCINFO where (DOC_GUID not in (select o_docguid from dms_doc)) order by IMPORT)
-> datasource: STRING(pwdsn_db0)
-> bmap.plan_id: STRING(63075731126d24faff9b56ca)
-> bmap.plan_name: STRING(hj-otel3)
{"kind": "exporter", "data_type": "traces", "name": "logging"}
2022-09-02T00:41:35.812Z error exporterhelper/queued_retry.go:183 **Exporting failed. The error is not retryable. Dropping data. {"kind": "exporter", "data_type": "traces", "name": "otlphttp/insecure_no_verify", "error": "Permanent error: error exporting items, request to https://apm.<removed>:8200/v1/traces responded with HTTP Status Code 400, Message=proto: illegal wireType 7, Details=[]", "dropped_items": 5**}
go.opentelemetry.io/collector/exporter/exporterhelper.(*retrySender).send
go.opentelemetry.io/collector@v0.56.0/exporter/exporterhelper/queued_retry.go:183
go.opentelemetry.io/collector/exporter/exporterhelper.(*tracesExporterWithObservability).send
go.opentelemetry.io/collector@v0.56.0/exporter/exporterhelper/traces.go:134
go.opentelemetry.io/collector/exporter/exporterhelper.(*queuedRetrySender).start.func1
go.opentelemetry.io/collector@v0.56.0/exporter/exporterhelper/queued_retry_inmemory.go:119
go.opentelemetry.io/collector/exporter/exporterhelper/internal.consumerFunc.consume
go.opentelemetry.io/collector@v0.56.0/exporter/exporterhelper/internal/bounded_memory_queue.go:82
go.opentelemetry.io/collector/exporter/exporterhelper/internal.(*boundedMemoryQueue).StartConsumers.func2
go.opentelemetry.io/collector@v0.56.0/exporter/exporterhelper/internal/bounded_memory_queue.go:69
2022-09-02T00:43:25.011Z info TracesExporter {"kind": "exporter", "data_type": "traces", "name": "logging", "#spans": 5}
2022-09-02T00:43:25.011Z info ResourceSpans #0
