Failed to push trace data via OTLP exporter: rpc error: code = Unavailable desc = connection closed

If you are asking about a problem you are experiencing, please use the following template, as it will help us help you. If you have a different problem, please delete all of this text :slight_smile:

TIP 1: select at least one tag that further categorizes your topic. For example server for APM Server related questions, java for questions regarding the Elastic APM Java agent, or ui for questions about the APM App within Kibana.
server

TIP 2: Check out the troubleshooting guide first. Not only will it help you to resolve common problems faster but it also explains in more detail which information we need before we can properly help you.

Kibana version: 7.15

Elasticsearch version: 7.15

APM Server version: 7.15

APM Agent language and version: Opentelemetry Agent

Browser version: NA

Original install method (e.g. download page, yum, deb, from source, etc.) and version:

Installed APM, Kibana and Elasticsearch on Kubernetes.

Fresh install or upgraded from other version?
Fresh

Is there anything special in your setup? For example, are you using the Logstash or Kafka outputs? Are you using a load balancer in front of the APM Servers? Have you changed index pattern, generated custom templates, changed agent configuration etc.
No

Description of the problem including expected versus actual behavior. Please include screenshots (if relevant):

Steps to reproduce:

  1. Followed the exact steps mentioned in Elastic Cloud on Kubernetes [1.8] | Elastic to install quickstart APM server, Kibana and Elasticsearch in AKS cluster.
  2. APM service is is listening on IP xx.xxx.xxx.xxx at port 8200
  3. I am also able to access Kibana through browser.
  4. Installed Opentelemetry collector and its exporter section is as mentioned below

exporters:
otlp:
endpoint: http://xx.xxx.xxx.xxx:8200
headers:
Authorization: "Bearer XXXXXXXXXXXX"
insecure: true

  1. Instrumented my application using Opentelemetry agent. Configured the agent to send data to OTLP collector.

  2. Please note: Application, APM server and Collector are installed in same namespace in AKS cluster.

The data is being sent to collector but collector is unable to send data to APM server. Getting error in OTLP collector.

error exporterhelper/queued_retry.go:304 Exporting failed. No more retries left. Dropping data. {"kind": "exporter", "name": "otlp", "error": "max elapsed time expired failed to push trace data via OTLP exporter: rpc error: code = Unavailable desc = connection closed", "dropped_items": 512}
go.opentelemetry.io/collector/exporter/exporterhelper.(*retrySender).send
go.opentelemetry.io/collector/exporter/exporterhelper/queued_retry.go:304
go.opentelemetry.io/collector/exporter/exporterhelper.(*tracesExporterWithObservability).send
go.opentelemetry.io/collector/exporter/exporterhelper/traces.go:116
go.opentelemetry.io/collector/exporter/exporterhelper.(*queuedRetrySender).start.func1
go.opentelemetry.io/collector/exporter/exporterhelper/queued_retry.go:155
go.opentelemetry.io/collector/exporter/exporterhelper/internal.ConsumerFunc.Consume
go.opentelemetry.io/collector/exporter/exporterhelper/internal/bounded_queue.go:103
go.opentelemetry.io/collector/exporter/exporterhelper/internal.(*BoundedQueue).StartConsumersWithFactory.func1
go.opentelemetry.io/collector/exporter/exporterhelper/internal/bounded_queue.go:82

I don't see any errors in APM server.

Errors in browser console (if relevant): NA

Provide logs and/or server output (if relevant):

After disabling TLS on APM server, the issue is resolved.
Hence this topic can be closed.

1 Like