serverMap Module data exceptions

tttoad · July 7, 2021, 3:51am

APM Server version 7.13.2 elasticSearch: 7.13.2 kibana: 7.13.2

The service node of the serverMap module on the Kibana Apm interface cannot be linked up.

1.run apm,es,kibana:

docker network create -d bridge my-jaeger-net
docker run --name elasticsearch --network=my-jaeger-net -p 9200:9200 -p 9300:9300 -e "discovery.type=single-node" -d elasticsearch:7.13.2
docker run --name kibana --network=my-jaeger-net -p 5601:5601 -d kibana:7.13.2
docker run -d -p 8200:8200 --name=apm-server --network=my-jaeger-net --user=apm-server elastic/apm-server:7.13.2 --strict.perms=false -e -E output.elasticsearch.hosts=["elasticsearch:9200"]

run openTelemetry: 0.27.0

	docker run  --name collector --network my-jaeger-net\
  		  -v otelcontribcol_config.yaml:/otelcontribcol_config.yaml \
          -d otel/opentelemetry-collector-contrib:0.27.0 --config=/config.yaml

config.yaml:

receivers:
  jaeger:
    protocols:
      grpc:
        endpoint: 0.0.0.0:14250
exporters:
  logging/detail:
    loglevel: debug
  alibabacloud_logservice/canary:
     /*******/
  elastic:
    apm_server_url: http://apm-server:8200

service:
  pipelines:
    traces:
      receivers: [ jaeger ] 
      exporters: [ alibabacloud_logservice/canary, elastic ]

3.run jaeger-agent:

    docker run -p 6831:6831/udp --network my-jaeger-net  -d jaegertracing/jaeger-agent:1.22.0 --reporter.grpc.host-port=collector:14250

Use uber/jaeger-client-go to send data to jaeger-agent,agent to send data to openTelemetry
server_t1:

package main
import (
	"fmt"
	"io"
	"log"
	"net/http"

	"github.com/opentracing/opentracing-go"
	"github.com/opentracing/opentracing-go/ext"
	"github.com/uber/jaeger-client-go"
	"github.com/uber/jaeger-client-go/config"
)

// service
type service struct {
}


func (m *MySpan) Context() opentracing.SpanContext {
	return m.SpanContext()
}
func (s *service) ServeHTTP(writer http.ResponseWriter, request *http.Request) {
	fmt.Println("request******")
	tracer := opentracing.GlobalTracer()
	spanCtx, err := tracer.Extract(opentracing.HTTPHeaders, opentracing.HTTPHeadersCarrier(request.Header))
	if err != nil {
		log.Fatal(err)
	}

	span := tracer.StartSpan("service_one_do", ext.RPCServerOption(spanCtx))
	span.Context()
	defer func() {
		span.Finish()
	}()

	writer.Write([]byte("12331"))
}

func main() {
	tracer, closer := initJaeger("service_t2")
	defer closer.Close()

	opentracing.SetGlobalTracer(tracer)
	fmt.Println(http.ListenAndServe(":8083", &service{}))
}

func initJaeger(service string) (opentracing.Tracer, io.Closer) {
	cfg := &config.Configuration{
		Sampler: &config.SamplerConfig{
			Type:  "const",
			Param: 1,
		},
		Reporter: &config.ReporterConfig{
			LogSpans:           true,
			LocalAgentHostPort: "127.0.0.1:6831",
		},
		ServiceName: service,
	}

	tracer, closer, err := cfg.NewTracer(config.Logger(jaeger.StdLogger))
	if err != nil {
		panic(fmt.Sprintf("ERROR: cannot init Jaeger: %v\n", err))
	}

	return tracer, closer
}

server-two:

package main

import (
	"bytes"
	"fmt"
	"io"
	"log"
	"net/http"

	"github.com/opentracing/opentracing-go"
	"github.com/opentracing/opentracing-go/ext"
	"github.com/uber/jaeger-client-go"
	"github.com/uber/jaeger-client-go/config"
)

func main() {
	tracer, closer := initJaeger("service_t1")

	defer closer.Close()

	opentracing.SetGlobalTracer(tracer)

	do("1")
	// do("2")
}
func do(i string) {
	var url = "http://127.0.0.1:8083"
	tracer := opentracing.GlobalTracer()
	request, err := http.NewRequest(http.MethodGet, url, bytes.NewBuffer([]byte("")))
	if err != nil {
		log.Fatal(err)
	}
	span := tracer.StartSpan("service_two_do" + i)
	defer span.Finish()
	ext.SpanKindRPCClient.Set(span)
	ext.HTTPUrl.Set(span, url)
	ext.HTTPMethod.Set(span, http.MethodGet)

	span.SetTag("traces", "1")
	err = tracer.Inject(span.Context(), opentracing.HTTPHeaders, opentracing.HTTPHeadersCarrier(request.Header))
	if err != nil {
		log.Fatal(err)
	}
	do, err := http.DefaultClient.Do(request)
	if err != nil {
		log.Fatal(err)
	}
	ext.HTTPStatusCode.Set(span, uint16(do.StatusCode))

	all, err := io.ReadAll(do.Body)
	fmt.Println(string(all), err)
}

func initJaeger(service string) (opentracing.Tracer, io.Closer) {
	cfg := &config.Configuration{
		Sampler: &config.SamplerConfig{
			Type:  "const",
			Param: 1,
		},
		Reporter: &config.ReporterConfig{
			LogSpans:           true,
			LocalAgentHostPort: "127.0.0.1:6831",
		},
		ServiceName: service,
	}

	tracer, closer, err := cfg.NewTracer(config.Logger(jaeger.StdLogger))
	if err != nil {
		panic(fmt.Sprintf("ERROR: cannot init Jaeger: %v\n", err))
	}

	return tracer, closer
}

The apm interface can view the tracing information normally, but there is no connection information between the services of the serverMap module. The data is reported to AliCloud sls service at the same time, and AliCloud can show the dependency of the service normally.
Everything is fine when using go-agent to report data

axw · July 8, 2021, 2:27am

@ttoad welcome to the forum! Thanks for providing all of these details to reproduce the issue, it's really helpful.

The issue here is that Elastic APM and Jaeger have a different data model: Elastic APM has distinct concepts of "transactions" and "spans", whereas Jaeger has only "spans". You can think of a transaction as like an entry-point span -- the first span in a service, for example an incoming HTTP request (like in service_t2).

When we translate from Jaeger's data model into Elastic APM's, root Jaeger spans become Elastic APM transactions. For example, the "client span" in service_t1 becomes a transaction. This is why the service map is not being formed correctly. In Elastic APM, the service map requires destination information that is included only in Elastic APM spans.

To resolve this, you can create an additional Jaeger span in service_t1. Something like this:

tracer := opentracing.GlobalTracer()

// This span becomes an Elastic APM transaction, because it is the trace root.
span := tracer.StartSpan("service_two_do" + i)
defer span.Finish()

var url = "http://127.0.0.1:8083"
request, err := http.NewRequest(http.MethodGet, url, bytes.NewBuffer([]byte("")))
if err != nil {
        log.Fatal(err)
}

// This becomes an Elastic APM span, because it is a child of the above span
// and has the "client" kind. Setting HTTP tags on this span will populate the
// service map and dependency list in the UI.
span = span.Tracer().StartSpan(url, opentracing.ChildOf(span.Context()))
defer span.Finish()
ext.SpanKindRPCClient.Set(span)
ext.HTTPUrl.Set(span, url)
ext.HTTPMethod.Set(span, http.MethodGet)

tttoad · July 8, 2021, 2:56am

Thank you for your answer, but I want the effect that service_t1 and service_t2 are connected, is there any way to do it? Is there a way to modify the openTelemetry or apm to achieve this effect?

tttoad · July 8, 2021, 3:09am

I know what to write, thanks again for your answer.

cyrilleleclerc · July 12, 2021, 2:01pm

Hello @tttoad ,

As discussed on How does the "jaeger client" data access "APM" via "OpenTelemetry"? , I recommend you to use the Elastic OpenTelemetry Protocol native support rather than the OpenTelemetry Collector Exporter for Elastic that has been deprecated few months ago.

See our documentation OpenTelemetry integration | APM Overview [7.13] | Elastic .

The collector configuration will look like:

receivers:
  jaeger:
    protocols:
      grpc:
        endpoint: 0.0.0.0:14250
exporters:
  logging/detail:
    loglevel: debug
  alibabacloud_logservice/canary:
     /*******/
  otlp/elastic:
    endpoint: "localhost:8200"
    insecure: true
    headers:
      Authorization: "Bearer my_secret_token"

processors:
  memory_limiter:
    check_interval: 1s
    limit_mib: 2000
  batch:

service:
  pipelines:
    traces:
      receivers: [ jaeger ] 
      processors: [ memory_limiter, batch ]
      exporters: [ alibabacloud_logservice/canary, otlp/elastic ]

tttoad · July 13, 2021, 2:49am

I see that the configuration method given in the documentation is still "elastic" instead of "otel/elastic".

cyrilleleclerc · July 13, 2021, 8:35am

Thanks, we will fix this ASAP.

system · August 3, 2021, 4:35am

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
How does the "jaeger client" data access "APM" via "OpenTelemetry"? APM docker , go , server	3	1644	August 5, 2021
No information in the "Metrics" module of the "APM" screen APM docker , go	5	1058	August 16, 2021
APM server producing errors (mapper_parsing_exception) when traces are sent through jaeger-agent APM docker , server	9	1553	October 5, 2020
APM Server has still not connected to Elasticsearch (2, previous was closed) APM	12	2248	December 11, 2019
Apm-server config in docker compose with ES and Kibana APM docker , server	4	4985	April 24, 2022

serverMap Module data exceptions

Related topics