How to integrate the "metrics" and "trace" data

tttoad · July 16, 2021, 2:20am

I want to integrate the "metrics" and "trace" data, the "metrics" data is grabbed by "prometheus". Now I use "openTelemetry" "receivers.prometheus_simple" to collect the data and send it to "APM The data is sent to "APM" and cannot be integrated with "trace". The "service.name" of both is different.

Step:
1.run apm,es,kibana:

docker network create -d bridge my-jaeger-net
docker run --name elasticsearch --network=my-jaeger-net -p 9200:9200 -p 9300:9300 -e "discovery.type=single-node" -d elasticsearch:7.13.2
docker run --name kibana --network=my-jaeger-net -p 5601:5601 -d kibana:7.13.2
docker run -d -p 8200:8200 --name=apm-server --network=my-jaeger-net --user=apm-server elastic/apm-server:7.13.2 --strict.perms=false -e -E output.elasticsearch.hosts=["elasticsearch:9200"]

run openTelemetry: 0.29.0

docker run  --name collector --network my-jaeger-net\
  		  -v otelcontribcol_config.yaml:/otelcontribcol_config.yaml \
          -d otel/opentelemetry-collector-contrib:0.27.0 --config=/config.yaml

config :

receivers:
  jaeger:
    protocols:
      grpc:
        endpoint: 0.0.0.0:14250
  prometheus_simple:
    collection_interval: 10s
    endpoint: "main2:9102"

processors:
  memory_limiter:
    check_interval: 1s
    limit_mib: 2000
  batch:
    
exporters:
  logging/detail:
    loglevel: debug

  otlp/elastic:
    endpoint: apm-server:8200
    insecure: true
    headers:
      Authorization: "Bearer my_secret_token"

service:
  pipelines:
    traces:
      receivers: [ jaeger ] #接收端配置为jaeger。
      exporters: [ otlp/elastic ] #发送端配置为alibabacloud_logservice/sls-trace。
      processors: [ memory_limiter, batch ]
    metrics:
      receivers: [ prometheus_simple ]
      exporters: [ otlp/elastic ]

3.run main2 service
main:9102/metrics response:

# HELP go_gc_duration_seconds A summary of the pause duration of garbage collection cycles.
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 5.87e-05
go_gc_duration_seconds{quantile="0.25"} 0.0001382
go_gc_duration_seconds{quantile="0.5"} 0.0003104
go_gc_duration_seconds{quantile="0.75"} 0.000651
go_gc_duration_seconds{quantile="1"} 0.0023786
go_gc_duration_seconds_sum 0.0035369
go_gc_duration_seconds_count 5

go_goroutines 260

go_info{version="go1.16.6"} 1

go_memstats_alloc_bytes 5.843192e+06

go_memstats_alloc_bytes_total 1.7327112e+07

go_memstats_buck_hash_sys_bytes 1.449204e+06

go_memstats_frees_total 26693

go_memstats_gc_cpu_fraction 0.01836957988419095

go_memstats_gc_sys_bytes 5.213408e+06

go_memstats_heap_alloc_bytes 5.843192e+06

go_memstats_heap_idle_bytes 5.8195968e+07

go_memstats_heap_inuse_bytes 7.53664e+06

go_memstats_heap_objects 24186

go_memstats_heap_released_bytes 5.709824e+07

go_memstats_heap_sys_bytes 6.5732608e+07

go_memstats_last_gc_time_seconds 1.6263290546848054e+09

go_memstats_lookups_total 0

go_memstats_mallocs_total 50879

go_memstats_mcache_inuse_bytes 7200

go_memstats_mcache_sys_bytes 16384

go_memstats_mspan_inuse_bytes 124440

go_memstats_mspan_sys_bytes 131072

go_memstats_next_gc_bytes 7.063104e+06

go_memstats_other_sys_bytes 1.399348e+06

go_memstats_stack_inuse_bytes 1.376256e+06

go_memstats_stack_sys_bytes 1.376256e+06

go_memstats_sys_bytes 7.531828e+07

go_threads 12

gorm_dbstats_idle{db="jianjiu",micro_name="lb.example.test11"} 0

gorm_dbstats_in_use{db="jianjiu",micro_name="lb.example.test11"} 0


gorm_dbstats_max_idle_closed{db="jianjiu",micro_name="lb.example.test11"} 0


gorm_dbstats_max_lifetime_closed{db="jianjiu",micro_name="lb.example.test11"} 0

gorm_dbstats_max_open_connections{db="jianjiu",micro_name="lb.example.test11"} 0


gorm_dbstats_open_connections{db="jianjiu",micro_name="lb.example.test11"} 0


gorm_dbstats_wait_count{db="jianjiu",micro_name="lb.example.test11"} 0

gorm_dbstats_wait_duration{db="jianjiu",micro_name="lb.example.test11"} 0

process_cpu_seconds_total 1.86

process_max_fds 1.048576e+06

process_open_fds 22

process_resident_memory_bytes 4.3569152e+07

process_start_time_seconds 1.62632905214e+09

process_virtual_memory_bytes 7.70568192e+08

process_virtual_memory_max_bytes 1.8446744073709552e+19

promhttp_metric_handler_requests_in_flight{micro_name="lb.example.test11"} 1

promhttp_metric_handler_requests_total{code="200",micro_name="lb.example.test11"} 3
promhttp_metric_handler_requests_total{code="500",micro_name="lb.example.test11"} 0
promhttp_metric_handler_requests_total{code="503",micro_name="lb.example.test11"} 0

The result is as follows: (I have only grabbed the "metrics" of "lb.example.test11" so far)

{
  "_index": "apm-7.13.3-metric-000001",
  "_type": "_doc",
  "_id": "xMzcqHoBcStLGm-E-rvj",
  "_version": 1,
  "_score": null,
  "fields": {
    "host.hostname": [
      "main2"
    ],
    "process_virtual_memory_bytes": [
      770830340
    ],
    "go_memstats_mspan_sys_bytes": [
      180224
    ],
    "go_memstats_last_gc_time_seconds": [
      1626330620
    ],
    "go_memstats_stack_sys_bytes": [
      1474560
    ],
    "service.language.name": [
      "unknown"
    ],
    "labels.instance": [
      "main2:9102"
    ],
    "scrape_samples_post_metric_relabeling": [
      92
    ],
    "labels.scheme": [
      "http"
    ],
    "processor.event": [
      "metric"
    ],
    "agent.name": [
      "otlp"
    ],
    "process_start_time_seconds": [
      1626330240
    ],
    "host.name": [
      "main2"
    ],
    "up": [
      1
    ],
    "go_memstats_heap_alloc_bytes": [
      9424888
    ],
    "go_memstats_mspan_inuse_bytes": [
      155448
    ],
    "go_memstats_lookups_total": [
      0
    ],
    "go_memstats_frees_total": [
      134718
    ],
    "go_memstats_alloc_bytes": [
      9424888
    ],
    "processor.name": [
      "metric"
    ],
    "go_memstats_sys_bytes": [
      75580424
    ],
    "go_memstats_buck_hash_sys_bytes": [
      1455916
    ],
    "go_memstats_mallocs_total": [
      174129
    ],
    "ecs.version": [
      "1.8.0"
    ],
    "observer.type": [
      "apm-server"
    ],
    "observer.version": [
      "7.13.3"
    ],
    "go_memstats_gc_cpu_fraction": [
      0.000038187492
    ],
    "agent.version": [
      "unknown"
    ],
    "scrape_duration_seconds": [
      0.0027382
    ],
    "process_resident_memory_bytes": [
      28372992
    ],
    "process_cpu_seconds_total": [
      70.78
    ],
    "service.node.name": [
      "main2"
    ],
    "go_memstats_heap_idle_bytes": [
      54247424
    ],
    "go_memstats_heap_sys_bytes": [
      65634304
    ],
    "go_memstats_gc_sys_bytes": [
      5462296
    ],
    "process_open_fds": [
      22
    ],
    "scrape_series_added": [
      92
    ],
    "go_memstats_stack_inuse_bytes": [
      1474560
    ],
    "process_max_fds": [
      1048576
    ],
    "go_memstats_heap_objects": [
      39411
    ],
    "go_memstats_mcache_inuse_bytes": [
      7200
    ],
    "go_memstats_other_sys_bytes": [
      1356740
    ],
    "go_threads": [
      12
    ],
    "go_memstats_heap_inuse_bytes": [
      11386880
    ],
    "go_memstats_mcache_sys_bytes": [
      16384
    ],
    "go_memstats_alloc_bytes_total": [
      37136680
    ],
    "labels.port": [
      "9102"
    ],
    "go_goroutines": [
      265
    ],
    "service.name": [
      "prometheus_simple_main2_9102"
    ],
    "labels.job": [
      "prometheus_simple/main2:9102"
    ],
    "go_memstats_heap_released_bytes": [
      49135616
    ],
    "observer.version_major": [
      7
    ],
    "process_virtual_memory_max_bytes": [
      18446744000000000000
    ],
    "observer.hostname": [
      "797c4982dd79"
    ],
    "scrape_samples_scraped": [
      92
    ],
    "metricset.name": [
      "app"
    ],
    "event.ingested": [
      "2021-07-15T06:31:32.316Z"
    ],
    "@timestamp": [
      "2021-07-15T06:31:31.343Z"
    ],
    "go_memstats_next_gc_bytes": [
      16465936
    ]
  },
  "sort": [
    1626330691343
  ]
}

cyrilleleclerc · July 20, 2021, 10:51am

Can you please explain us

Where the traces come from: application type, instrumentation type... The receiver seems to be Jaeger for traces and your screenshot indicates a "lb" which probably stands for "load balancer"
Where the prometheus metrics come from: what type of prometheus exporter do you use for the metrics...

Note that some system that are not yet supporting instrumentation with OpenTelemetry support OpenCensus in addition to Jaeger and may, thanks to OpenCensus, export metrics in addition to traces.

For your prometheus scrapper configuration, did you consider defining the label service.name using the external_labels capability of the scrape_config configuration? See docs prometheus/configuration.md at v2.28.1 · prometheus/prometheus · GitHub

tttoad · July 20, 2021, 12:37pm

The "trace" comes from "uber-jaeger-client -> jaeger-agent -> openTelemetry".
"trace" information:

The "metric" is crawling the service's port 9102 through the "prometheus_simple" of "openTelemetry". The service exposes "metrics" data via "prometheus/client_golang/prometheus".

import 	(
       "github.com/prometheus/client_golang/prometheus/promhttp"
       "net/http"
)
http.Handle("/metrics", promhttp.Handler())

The "lb" is our company abbreviation.

cyrilleleclerc · July 20, 2021, 12:46pm

Thanks @tttoad,

Did you consider switching from the Jaeger client (traces) + Prometheus clients (metrics) to an OpenTelemetry client that would handle both traces and metrics?

If you don't want to or can't change the instrumentation library, I would evaluate switching on the Otel Collector from the prometheus_simplereceiver to the more advanced prometheus receiver and add a label in the scrape_config using the external_labels.

Would this pattern make sense to you?

tttoad · July 20, 2021, 1:13pm

Thank you for your answer!
Do you already support the "metrics" data reported by the "OpenTelemetry-client" ? We will consider switching to "OpenTelemetry-client" for reporting information...

cyrilleleclerc · July 20, 2021, 2:42pm

Yes, we have been supporting Otel Metrics for a while, the last missing bit is to support Histograms and we should fix it soon. Note that you can instrument your applications with OpenTelemetry agents or SDKs and use an OpenTelemetry Collector next to your applications to push both to Elastic Observability, to Prometheus and to any other destination.

See our documentation OpenTelemetry integration | APM User Guide [8.11] | Elastic

system · August 10, 2021, 10:42am

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
No information in the "Metrics" module of the "APM" screen APM docker , go	5	1058	August 16, 2021
Opentery reported the data from Trace, Metrics data at Prometheus. How to connect? APM ui	1	368	August 3, 2021
How to connect prometheus metrics with openTelemtry report trace data APM elastic-stack-monitoring , go	1	531	July 30, 2021
ELK APM and Jaeger Integration add Kubernetes metadata and system data to transactions/spans APM docker , server	2	417	February 23, 2021
OpenTelemetry data is received by APM Server but not stored in Elasticsearch Elastic Observability docker	0	278	December 19, 2022

How to integrate the "metrics" and "trace" data

Related topics