Error while dialing: dial tcp: lookup apm-server on 127.0.0.11:53

[Hamad (Muhammad Hammad)]

I have installed apm-server to show traces and metrics that are being exported from otel. I can see my service on the APM on kibana but cant see the traces that are exported by otel. while seeing logs of otel collector on docker i found following log:

 info    exporterhelper/queued_retry.go:430      Exporting failed. Will retry the request after interval.       {"kind": "exporter", "data_type": "traces", "name": "otlp/elastic", "error": "rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp: lookup apm-server on 127.0.0.11:53: no such host\"", "interval": "6.545859934s"}

warn    zapgrpc/zapgrpc.go:195  [core] [Channel #6 SubChannel #7] grpc: addrConn.createTransport failed to connect to {Addr: "apm-server:8200", ServerName: "apm-server:8200", }. Err: connection error: desc = "transport: Error while dialing: dial tcp: lookup apm-server on 127.0.0.11:53: no such host"    {"grpc_log": true}

my apm-server.yml file:

output.elasticsearch:
  hosts: ['localhost:9200']
  username: elastic
  password: changeme

my otel config file:

exporters:
  otlp:
    endpoint: 'http://localhost:8200'
    tls:
      insecure: true
    headers:
      Authorization: 'Bearer '

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [zipkin, logging, otlp]
    metrics:
      receivers: [otlp]
      exporters: [logging, otlp]
    logs:
      receivers: [otlp]
      exporters: [logging, otlp]

[lahsivjar (Vishal Raj)]

Hello @Hamad ,

I can see my service on the APM on kibana

Are you able to see metrics exported from OTEL in kibana?

while seeing logs of otel collector on docker i found following log

It looks like your OTEL collector cannot access APM-Server. I see in the configuration that you are using localhost:8200 as the OTLP endpoint, I don't think the OTEL collector running on docker can reach the APM-Server on localhost no matter where it is running without some special handling. Can you confirm that APM-Server is reachable from OTEL collector docker container?

[Hamad (Muhammad Hammad)]

Hey @lahsivjar , thanks for replying. I was facing multiple issues with setting up apm server locally so shifted to elastic cloud. It was a lot easier to setup apm on cloud.

[lahsivjar (Vishal Raj)]

Glad to hear that