Sending traces from the Jaeger hotrod app to Elastic APM on Openshift

Elastic Observability APM
1

If you are asking about a problem you are experiencing, please use the following template, as it will help us help you. If you have a different problem, please delete all of this text :slight_smile:

Kibana version: 7.6.1

Elasticsearch version: 7.6.1

APM Server version: 7.6.1

APM Agent language and version: n/A

Browser version: Chrome Version 80.0.3987.122 (Official Build) (32-bit)

Original install method (e.g. download page, yum, deb, from source, etc.) and version: helm chart deployments with custom images (using official docker file with a different base)

Fresh install or upgraded from other version? Fresh

Is there anything special in your setup? For example, are you using the Logstash or Kafka outputs? Are you using a load balancer in front of the APM Servers? Have you changed index pattern, generated custom templates, changed agent configuration etc.

Description of the problem including expected versus actual behavior. Please include screenshots (if relevant):

On Openshift 3.6, I am using Jaeger to send traces to Elasticsearch vi the collector and ingestor services. I would like to see how these appear on the APM Server. I tried running the Jaeger hotrod app to send traces to the APM Server running within the same Openshift namespace. The jaeger ports are exposed via a service. This approach works when the hot rod app sends traces to the Jaeger collector using the Jaeger endpoint. However, when swapped with the apm server endpoint, the app throws errors:

|2020-04-15T23:48:28.602Z|ERROR|log/logger.go:43|error reporting Jaeger span "FindDriverIDs": Post http://apm-server-dev..svc.cluster.local:14268/api/traces: dial tcp 172.30.138.225:14268: connect: no route to host|{"service": "driver"}|
|2020-04-15T23:48:28.602Z|ERROR|log/logger.go:43|error reporting Jaeger span "HTTP GET /": Post http://apm-server-dev..svc.cluster.local:14268/api/traces: dial tcp 172.30.138.225:14268: connect: no route to host|{"service": "frontend"}|
|2020-04-15T23:48:28.602Z|ERROR|log/logger.go:43|error reporting Jaeger span "HTTP GET /route": Post http://apm-server-dev..svc.cluster.local:14268/api/traces: dial tcp 172.30.138.225:14268: connect: no route to host|{"service": "route"}|
|2020-04-15T23:48:28.602Z|ERROR|log/logger.go:43|error reporting Jaeger span "Driver::findNearest": Post http://apm-server-dev..svc.cluster.local:14268/api/traces: dial tcp 172.30.138.225:14268: connect: no route to host|{"service": "driver"}|
|2020-04-15T23:48:28.602Z|ERROR|log/logger.go:43|error reporting Jaeger span "SQL SELECT": Post http://apm-server-dev..svc.cluster.local:14268/api/traces: dial tcp 172.30.138.225:14268: connect: no route to host|{"service": "customer"}|
|2020-04-15T23:48:28.602Z|ERROR|log/logger.go:43|error reporting Jaeger span "HTTP GET /customer": Post http://apm-server-dev..svc.cluster.local:14268/api/traces: dial tcp 172.30.138.225:14268: connect: no route to host|{"service": "customer"}|
|2020-04-15T23:48:31.608Z|ERROR|log/logger.go:43|error reporting Jaeger span "SQL SELECT": Post http://apm-server-dev..svc.cluster.local:14268/api/traces: dial tcp 172.30.138.225:14268: connect: no route to host|{"service": "customer"}|
|2020-04-15T23:48:31.608Z|ERROR|log/logger.go:43|error reporting Jaeger span "HTTP GET /": Post http://apm-server-dev..svc.cluster.local:14268/api/traces: dial tcp 172.30.138.225:14268: connect: no route to host|{"service": "frontend"}|
|2020-04-15T23:48:31.608Z|ERROR|log/logger.go:43|error reporting Jaeger span "GetDriver": Post http://apm-server-dev..svc.cluster.local:14268/api/traces: dial tcp 172.30.138.225:14268: connect: no route to host|{"service": "driver"}|
|2020-04-15T23:48:31.608Z|ERROR|log/logger.go:43|error reporting Jaeger span "Driver::findNearest": Post http://apm-server-dev..svc.cluster.local:14268/api/traces: dial tcp 172.30.138.225:14268: connect: no route to host|{"service": "driver"}|
|2020-04-15T23:48:31.608Z|ERROR|log/logger.go:43|error reporting Jaeger span "HTTP GET /customer": Post http://apm-server-dev..svc.cluster.local:14268/api/traces: dial tcp 172.30.138.225:14268: connect: no route to host|{"service": "customer"}|
|2020-04-15T23:48:31.608Z|ERROR|log/logger.go:43|error reporting Jaeger span "HTTP GET /route": Post http://apm-server-dev..svc.cluster.local:14268/api/traces: dial tcp 172.30.138.225:14268: connect: no route to host|{"service": "route"}|

I've removed reference to the namespace.

NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
svc/apm-server-dev 172.30.138.225 8200/TCP,14250/TCP,14268/TCP

Steps to reproduce:

  1. Enable porst on apm-server for jaeger via configmap or apm-server.yaml
  2. Run the hotrod app on openshift using the helmcharts or openshift templates.
  3. Point to the apm server jaeger endpoint via a local service
  4. Trigger traces to be generated on the hot rod via the front end gui
2

Hi @sayeedc, welcome to the forum!

Have you enabled the Jaeger HTTP/Thrift endpoint in apm-server? And configured it to listen on all interfaces? You should have configuration like:

apm-server.jaeger.http.enabled: true # default is false
apm-server.jaeger.http.host: ":14268" # default is "localhost:14268"
3

Here are my settings:

apm-server:
  jaeger:
     grpc:
         enabled: true
         host: "0.0.0.0:14250"
     http:
         enabled: true
         host: "0.0.0.0:14268"
4

Here are settings on the Kubernetes service:

spec:
  clusterIP: 172.30.138.225
  ports:
  - name: http
    port: 8200
    protocol: TCP
    targetPort: 8200
  - name: jaeger-grpc
    port: 14250
    protocol: TCP
    targetPort: jaeger-grpc
  - name: jaeger-http
    port: 14268
    protocol: TCP
    targetPort: jaeger-http
5

I'm using the helm chart and template from here:

On the configmap, I've also added in the extra ports for Jaeger.

6

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.