serverMap Module data exceptions

APM Server version 7.13.2 elasticSearch: 7.13.2 kibana: 7.13.2

The service node of the serverMap module on the Kibana Apm interface cannot be linked up.

1.run apm,es,kibana:

docker network create -d bridge my-jaeger-net
docker run --name elasticsearch --network=my-jaeger-net -p 9200:9200 -p 9300:9300 -e "discovery.type=single-node" -d elasticsearch:7.13.2
docker run --name kibana --network=my-jaeger-net -p 5601:5601 -d kibana:7.13.2
docker run -d -p 8200:8200 --name=apm-server --network=my-jaeger-net --user=apm-server elastic/apm-server:7.13.2 --strict.perms=false -e -E output.elasticsearch.hosts=["elasticsearch:9200"]
  1. run openTelemetry: 0.27.0
	docker run  --name collector --network my-jaeger-net\
  		  -v otelcontribcol_config.yaml:/otelcontribcol_config.yaml \
          -d otel/opentelemetry-collector-contrib:0.27.0 --config=/config.yaml

config.yaml:

receivers:
  jaeger:
    protocols:
      grpc:
        endpoint: 0.0.0.0:14250
exporters:
  logging/detail:
    loglevel: debug
  alibabacloud_logservice/canary:
     /*******/
  elastic:
    apm_server_url: http://apm-server:8200

service:
  pipelines:
    traces:
      receivers: [ jaeger ] 
      exporters: [ alibabacloud_logservice/canary, elastic ]

3.run jaeger-agent:

    docker run -p 6831:6831/udp --network my-jaeger-net  -d jaegertracing/jaeger-agent:1.22.0 --reporter.grpc.host-port=collector:14250
  1. Use uber/jaeger-client-go to send data to jaeger-agent,agent to send data to openTelemetry
    server_t1:
package main
import (
	"fmt"
	"io"
	"log"
	"net/http"

	"github.com/opentracing/opentracing-go"
	"github.com/opentracing/opentracing-go/ext"
	"github.com/uber/jaeger-client-go"
	"github.com/uber/jaeger-client-go/config"
)

// service
type service struct {
}


func (m *MySpan) Context() opentracing.SpanContext {
	return m.SpanContext()
}
func (s *service) ServeHTTP(writer http.ResponseWriter, request *http.Request) {
	fmt.Println("request******")
	tracer := opentracing.GlobalTracer()
	spanCtx, err := tracer.Extract(opentracing.HTTPHeaders, opentracing.HTTPHeadersCarrier(request.Header))
	if err != nil {
		log.Fatal(err)
	}

	span := tracer.StartSpan("service_one_do", ext.RPCServerOption(spanCtx))
	span.Context()
	defer func() {
		span.Finish()
	}()

	writer.Write([]byte("12331"))
}

func main() {
	tracer, closer := initJaeger("service_t2")
	defer closer.Close()

	opentracing.SetGlobalTracer(tracer)
	fmt.Println(http.ListenAndServe(":8083", &service{}))
}

func initJaeger(service string) (opentracing.Tracer, io.Closer) {
	cfg := &config.Configuration{
		Sampler: &config.SamplerConfig{
			Type:  "const",
			Param: 1,
		},
		Reporter: &config.ReporterConfig{
			LogSpans:           true,
			LocalAgentHostPort: "127.0.0.1:6831",
		},
		ServiceName: service,
	}

	tracer, closer, err := cfg.NewTracer(config.Logger(jaeger.StdLogger))
	if err != nil {
		panic(fmt.Sprintf("ERROR: cannot init Jaeger: %v\n", err))
	}

	return tracer, closer
}

server-two:

package main

import (
	"bytes"
	"fmt"
	"io"
	"log"
	"net/http"

	"github.com/opentracing/opentracing-go"
	"github.com/opentracing/opentracing-go/ext"
	"github.com/uber/jaeger-client-go"
	"github.com/uber/jaeger-client-go/config"
)

func main() {
	tracer, closer := initJaeger("service_t1")

	defer closer.Close()

	opentracing.SetGlobalTracer(tracer)

	do("1")
	// do("2")
}
func do(i string) {
	var url = "http://127.0.0.1:8083"
	tracer := opentracing.GlobalTracer()
	request, err := http.NewRequest(http.MethodGet, url, bytes.NewBuffer([]byte("")))
	if err != nil {
		log.Fatal(err)
	}
	span := tracer.StartSpan("service_two_do" + i)
	defer span.Finish()
	ext.SpanKindRPCClient.Set(span)
	ext.HTTPUrl.Set(span, url)
	ext.HTTPMethod.Set(span, http.MethodGet)

	span.SetTag("traces", "1")
	err = tracer.Inject(span.Context(), opentracing.HTTPHeaders, opentracing.HTTPHeadersCarrier(request.Header))
	if err != nil {
		log.Fatal(err)
	}
	do, err := http.DefaultClient.Do(request)
	if err != nil {
		log.Fatal(err)
	}
	ext.HTTPStatusCode.Set(span, uint16(do.StatusCode))

	all, err := io.ReadAll(do.Body)
	fmt.Println(string(all), err)
}

func initJaeger(service string) (opentracing.Tracer, io.Closer) {
	cfg := &config.Configuration{
		Sampler: &config.SamplerConfig{
			Type:  "const",
			Param: 1,
		},
		Reporter: &config.ReporterConfig{
			LogSpans:           true,
			LocalAgentHostPort: "127.0.0.1:6831",
		},
		ServiceName: service,
	}

	tracer, closer, err := cfg.NewTracer(config.Logger(jaeger.StdLogger))
	if err != nil {
		panic(fmt.Sprintf("ERROR: cannot init Jaeger: %v\n", err))
	}

	return tracer, closer
}
  1. The apm interface can view the tracing information normally, but there is no connection information between the services of the serverMap module. The data is reported to AliCloud sls service at the same time, and AliCloud can show the dependency of the service normally.

  2. Everything is fine when using go-agent to report data

@ttoad welcome to the forum! Thanks for providing all of these details to reproduce the issue, it's really helpful.

The issue here is that Elastic APM and Jaeger have a different data model: Elastic APM has distinct concepts of "transactions" and "spans", whereas Jaeger has only "spans". You can think of a transaction as like an entry-point span -- the first span in a service, for example an incoming HTTP request (like in service_t2).

When we translate from Jaeger's data model into Elastic APM's, root Jaeger spans become Elastic APM transactions. For example, the "client span" in service_t1 becomes a transaction. This is why the service map is not being formed correctly. In Elastic APM, the service map requires destination information that is included only in Elastic APM spans.

To resolve this, you can create an additional Jaeger span in service_t1. Something like this:

tracer := opentracing.GlobalTracer()

// This span becomes an Elastic APM transaction, because it is the trace root.
span := tracer.StartSpan("service_two_do" + i)
defer span.Finish()

var url = "http://127.0.0.1:8083"
request, err := http.NewRequest(http.MethodGet, url, bytes.NewBuffer([]byte("")))
if err != nil {
        log.Fatal(err)
}

// This becomes an Elastic APM span, because it is a child of the above span
// and has the "client" kind. Setting HTTP tags on this span will populate the
// service map and dependency list in the UI.
span = span.Tracer().StartSpan(url, opentracing.ChildOf(span.Context()))
defer span.Finish()
ext.SpanKindRPCClient.Set(span)
ext.HTTPUrl.Set(span, url)
ext.HTTPMethod.Set(span, http.MethodGet)

1 Like

Thank you for your answer, but I want the effect that service_t1 and service_t2 are connected, is there any way to do it? Is there a way to modify the openTelemetry or apm to achieve this effect?

I know what to write, thanks again for your answer.

1 Like

Hello @tttoad ,

As discussed on How does the "jaeger client" data access "APM" via "OpenTelemetry"? , I recommend you to use the Elastic OpenTelemetry Protocol native support rather than the OpenTelemetry Collector Exporter for Elastic that has been deprecated few months ago.

See our documentation OpenTelemetry integration | APM Overview [7.13] | Elastic .

The collector configuration will look like:

receivers:
  jaeger:
    protocols:
      grpc:
        endpoint: 0.0.0.0:14250
exporters:
  logging/detail:
    loglevel: debug
  alibabacloud_logservice/canary:
     /*******/
  otlp/elastic:
    endpoint: "localhost:8200"
    insecure: true
    headers:
      Authorization: "Bearer my_secret_token"

processors:
  memory_limiter:
    check_interval: 1s
    limit_mib: 2000
  batch:

service:
  pipelines:
    traces:
      receivers: [ jaeger ] 
      processors: [ memory_limiter, batch ]
      exporters: [ alibabacloud_logservice/canary, otlp/elastic ]


I see that the configuration method given in the documentation is still "elastic" instead of "otel/elastic".

Thanks, we will fix this ASAP.