# How to integrate the "metrics" and "trace" data
**URL:** https://discuss.elastic.co/t/how-to-integrate-the-metrics-and-trace-data/278841
**Category:** APM
**Tags:** docker, go, server
**Created:** [July 16, 2021, 2:20am UTC](https://discuss.elastic.co/t/how-to-integrate-the-metrics-and-trace-data/278841 "2021-07-16T02:20:35Z")
**Posts on this page:** 7
**Page:** 1
### Author:  [@tttoad](https://discuss.elastic.co/u/tttoad)
#### Post date: [July 16, 2021, 2:20am UTC](https://discuss.elastic.co/t/how-to-integrate-the-metrics-and-trace-data/278841/1 "2021-07-16T02:20:35Z")
I want to integrate the "metrics" and "trace" data, the "metrics" data is grabbed by "prometheus". Now I use "openTelemetry" "receivers.prometheus\_simple" to collect the data and send it to "APM The data is sent to "APM" and cannot be integrated with "trace". The "service.name" of both is different.
Step:
1.run apm,es,kibana:
```auto
docker network create -d bridge my-jaeger-net
docker run --name elasticsearch --network=my-jaeger-net -p 9200:9200 -p 9300:9300 -e "discovery.type=single-node" -d elasticsearch:7.13.2
docker run --name kibana --network=my-jaeger-net -p 5601:5601 -d kibana:7.13.2
docker run -d -p 8200:8200 --name=apm-server --network=my-jaeger-net --user=apm-server elastic/apm-server:7.13.2 --strict.perms=false -e -E output.elasticsearch.hosts=["elasticsearch:9200"]
```
1. run openTelemetry: 0.29.0
```auto
docker run --name collector --network my-jaeger-net\
-v otelcontribcol_config.yaml:/otelcontribcol_config.yaml \
-d otel/opentelemetry-collector-contrib:0.27.0 --config=/config.yaml
```
config :
```auto
receivers:
jaeger:
protocols:
grpc:
endpoint: 0.0.0.0:14250
prometheus_simple:
collection_interval: 10s
endpoint: "main2:9102"
processors:
memory_limiter:
check_interval: 1s
limit_mib: 2000
batch:
exporters:
logging/detail:
loglevel: debug
otlp/elastic:
endpoint: apm-server:8200
insecure: true
headers:
Authorization: "Bearer my_secret_token"
service:
pipelines:
traces:
receivers: [jaeger] #接收端配置为jaeger。
exporters: [otlp/elastic] #发送端配置为alibabacloud_logservice/sls-trace。
processors: [memory_limiter, batch]
metrics:
receivers: [prometheus_simple]
exporters: [otlp/elastic]
```
3.run main2 service
main:9102/metrics response:
```auto
# HELP go_gc_duration_seconds A summary of the pause duration of garbage collection cycles.
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 5.87e-05
go_gc_duration_seconds{quantile="0.25"} 0.0001382
go_gc_duration_seconds{quantile="0.5"} 0.0003104
go_gc_duration_seconds{quantile="0.75"} 0.000651
go_gc_duration_seconds{quantile="1"} 0.0023786
go_gc_duration_seconds_sum 0.0035369
go_gc_duration_seconds_count 5
go_goroutines 260
go_info{version="go1.16.6"} 1
go_memstats_alloc_bytes 5.843192e+06
go_memstats_alloc_bytes_total 1.7327112e+07
go_memstats_buck_hash_sys_bytes 1.449204e+06
go_memstats_frees_total 26693
go_memstats_gc_cpu_fraction 0.01836957988419095
go_memstats_gc_sys_bytes 5.213408e+06
go_memstats_heap_alloc_bytes 5.843192e+06
go_memstats_heap_idle_bytes 5.8195968e+07
go_memstats_heap_inuse_bytes 7.53664e+06
go_memstats_heap_objects 24186
go_memstats_heap_released_bytes 5.709824e+07
go_memstats_heap_sys_bytes 6.5732608e+07
go_memstats_last_gc_time_seconds 1.6263290546848054e+09
go_memstats_lookups_total 0
go_memstats_mallocs_total 50879
go_memstats_mcache_inuse_bytes 7200
go_memstats_mcache_sys_bytes 16384
go_memstats_mspan_inuse_bytes 124440
go_memstats_mspan_sys_bytes 131072
go_memstats_next_gc_bytes 7.063104e+06
go_memstats_other_sys_bytes 1.399348e+06
go_memstats_stack_inuse_bytes 1.376256e+06
go_memstats_stack_sys_bytes 1.376256e+06
go_memstats_sys_bytes 7.531828e+07
go_threads 12
gorm_dbstats_idle{db="jianjiu",micro_name="lb.example.test11"} 0
gorm_dbstats_in_use{db="jianjiu",micro_name="lb.example.test11"} 0
gorm_dbstats_max_idle_closed{db="jianjiu",micro_name="lb.example.test11"} 0
gorm_dbstats_max_lifetime_closed{db="jianjiu",micro_name="lb.example.test11"} 0
gorm_dbstats_max_open_connections{db="jianjiu",micro_name="lb.example.test11"} 0
gorm_dbstats_open_connections{db="jianjiu",micro_name="lb.example.test11"} 0
gorm_dbstats_wait_count{db="jianjiu",micro_name="lb.example.test11"} 0
gorm_dbstats_wait_duration{db="jianjiu",micro_name="lb.example.test11"} 0
process_cpu_seconds_total 1.86
process_max_fds 1.048576e+06
process_open_fds 22
process_resident_memory_bytes 4.3569152e+07
process_start_time_seconds 1.62632905214e+09
process_virtual_memory_bytes 7.70568192e+08
process_virtual_memory_max_bytes 1.8446744073709552e+19
promhttp_metric_handler_requests_in_flight{micro_name="lb.example.test11"} 1
promhttp_metric_handler_requests_total{code="200",micro_name="lb.example.test11"} 3
promhttp_metric_handler_requests_total{code="500",micro_name="lb.example.test11"} 0
promhttp_metric_handler_requests_total{code="503",micro_name="lb.example.test11"} 0
```
The result is as follows: (I have only grabbed the "metrics" of "lb.example.test11" so far)
```auto
{
"_index": "apm-7.13.3-metric-000001",
"_type": "_doc",
"_id": "xMzcqHoBcStLGm-E-rvj",
"_version": 1,
"_score": null,
"fields": {
"host.hostname": [
"main2"
],
"process_virtual_memory_bytes": [
770830340
],
"go_memstats_mspan_sys_bytes": [
180224
],
"go_memstats_last_gc_time_seconds": [
1626330620
],
"go_memstats_stack_sys_bytes": [
1474560
],
"service.language.name": [
"unknown"
],
"labels.instance": [
"main2:9102"
],
"scrape_samples_post_metric_relabeling": [
92
],
"labels.scheme": [
"http"
],
"processor.event": [
"metric"
],
"agent.name": [
"otlp"
],
"process_start_time_seconds": [
1626330240
],
"host.name": [
"main2"
],
"up": [
1
],
"go_memstats_heap_alloc_bytes": [
9424888
],
"go_memstats_mspan_inuse_bytes": [
155448
],
"go_memstats_lookups_total": [
0
],
"go_memstats_frees_total": [
134718
],
"go_memstats_alloc_bytes": [
9424888
],
"processor.name": [
"metric"
],
"go_memstats_sys_bytes": [
75580424
],
"go_memstats_buck_hash_sys_bytes": [
1455916
],
"go_memstats_mallocs_total": [
174129
],
"ecs.version": [
"1.8.0"
],
"observer.type": [
"apm-server"
],
"observer.version": [
"7.13.3"
],
"go_memstats_gc_cpu_fraction": [
0.000038187492
],
"agent.version": [
"unknown"
],
"scrape_duration_seconds": [
0.0027382
],
"process_resident_memory_bytes": [
28372992
],
"process_cpu_seconds_total": [
70.78
],
"service.node.name": [
"main2"
],
"go_memstats_heap_idle_bytes": [
54247424
],
"go_memstats_heap_sys_bytes": [
65634304
],
"go_memstats_gc_sys_bytes": [
5462296
],
"process_open_fds": [
22
],
"scrape_series_added": [
92
],
"go_memstats_stack_inuse_bytes": [
1474560
],
"process_max_fds": [
1048576
],
"go_memstats_heap_objects": [
39411
],
"go_memstats_mcache_inuse_bytes": [
7200
],
"go_memstats_other_sys_bytes": [
1356740
],
"go_threads": [
12
],
"go_memstats_heap_inuse_bytes": [
11386880
],
"go_memstats_mcache_sys_bytes": [
16384
],
"go_memstats_alloc_bytes_total": [
37136680
],
"labels.port": [
"9102"
],
"go_goroutines": [
265
],
"service.name": [
"prometheus_simple_main2_9102"
],
"labels.job": [
"prometheus_simple/main2:9102"
],
"go_memstats_heap_released_bytes": [
49135616
],
"observer.version_major": [
7
],
"process_virtual_memory_max_bytes": [
18446744000000000000
],
"observer.hostname": [
"797c4982dd79"
],
"scrape_samples_scraped": [
92
],
"metricset.name": [
"app"
],
"event.ingested": [
"2021-07-15T06:31:32.316Z"
],
"@timestamp": [
"2021-07-15T06:31:31.343Z"
],
"go_memstats_next_gc_bytes": [
16465936
]
},
"sort": [
1626330691343
]
}
```
---
### Author:  [@cyrilleleclerc](https://discuss.elastic.co/u/cyrilleleclerc)
#### Post date: [July 20, 2021, 10:51am UTC](https://discuss.elastic.co/t/how-to-integrate-the-metrics-and-trace-data/278841/2 "2021-07-20T10:51:27Z")
Can you please explain us
- Where the traces come from: application type, instrumentation type... The receiver seems to be Jaeger for traces and your screenshot indicates a "lb" which probably stands for "load balancer"
- Where the prometheus metrics come from: what type of prometheus exporter do you use for the metrics...
Note that some system that are not yet supporting instrumentation with OpenTelemetry support OpenCensus in addition to Jaeger and may, thanks to OpenCensus, export metrics in addition to traces.
For your prometheus scrapper configuration, did you consider defining the label `service.name` using the `external_labels` capability of the `scrape_config` configuration? See docs [prometheus/configuration.md at v2.28.1 · prometheus/prometheus · GitHub](https://github.com/prometheus/prometheus/blob/v2.28.1/docs/configuration/configuration.md#scrape_config)
---
### Author:  [@tttoad](https://discuss.elastic.co/u/tttoad)
#### Post date: [July 20, 2021, 12:37pm UTC](https://discuss.elastic.co/t/how-to-integrate-the-metrics-and-trace-data/278841/3 "2021-07-20T12:37:53Z")
The "trace" comes from "uber-jaeger-client -\> jaeger-agent -\> openTelemetry".
"trace" information:
> [@How does the "jaeger client" data access "APM" via "OpenTelemetry"?](https://discuss.elastic.co/t/how-does-the-jaeger-client-data-access-apm-via-opentelemetry/278375):
>
> APM Server version 7.13.2 elasticSearch: 7.13.2 kibana: 7.13.2 Questions: 1.The error rate module in the Overview screen shows "No data to display", and the "Error rate" in the Transactions screen is N/A. How do I set this value? 2.The "server Map" looks normal, but the "Dependencies" of the "Overview" shows "No items found" 3.How do I set the "span type"? All the modules related to it show "No data to display" Step: 1.run apm,es,kibana: docker network create -d bridge my-jaeger-net docke…
The "metric" is crawling the service's port 9102 through the "prometheus\_simple" of "openTelemetry". The service exposes "metrics" data via "prometheus/client\_golang/prometheus".
```auto
import (
"github.com/prometheus/client_golang/prometheus/promhttp"
"net/http"
)
http.Handle("/metrics", promhttp.Handler())
```
The "lb" is our company abbreviation.
---
### Author:  [@cyrilleleclerc](https://discuss.elastic.co/u/cyrilleleclerc)
#### Post date: [July 20, 2021, 12:46pm UTC](https://discuss.elastic.co/t/how-to-integrate-the-metrics-and-trace-data/278841/4 "2021-07-20T12:46:25Z")
Thanks @tttoad,
Did you consider switching from the Jaeger client (traces) + Prometheus clients (metrics) to an OpenTelemetry client that would handle both traces and metrics?
If you don't want to or can't change the instrumentation library, I would evaluate switching on the Otel Collector from the `prometheus_simple`receiver to the more advanced `prometheus` receiver and add a label in the `scrape_config` using the `external_labels`.
Would this pattern make sense to you?
---
### Author:  [@tttoad](https://discuss.elastic.co/u/tttoad)
#### Post date: [July 20, 2021, 1:13pm UTC](https://discuss.elastic.co/t/how-to-integrate-the-metrics-and-trace-data/278841/5 "2021-07-20T13:13:09Z")
Thank you for your answer!
Do you already support the "metrics" data reported by the "OpenTelemetry-client" ? We will consider switching to "OpenTelemetry-client" for reporting information...
---
### Author:  [@cyrilleleclerc](https://discuss.elastic.co/u/cyrilleleclerc)
#### Post date: [July 20, 2021, 2:42pm UTC](https://discuss.elastic.co/t/how-to-integrate-the-metrics-and-trace-data/278841/6 "2021-07-20T14:42:44Z")
> [@tttoad](#):
>
> Do you already support the "metrics" data reported by the "OpenTelemetry-client" ? We will consider switching to "OpenTelemetry-client" for reporting information...
Yes, we have been supporting Otel Metrics for a while, the last missing bit is to support Histograms and we should fix it soon. Note that you can instrument your applications with OpenTelemetry agents or SDKs and use an OpenTelemetry Collector next to your applications to push both to Elastic Observability, to Prometheus and to any other destination.
See our documentation [OpenTelemetry integration | APM User Guide [8.11] | Elastic](https://www.elastic.co/guide/en/apm/get-started/current/open-telemetry-elastic.html)
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [August 10, 2021, 10:42am UTC](https://discuss.elastic.co/t/how-to-integrate-the-metrics-and-trace-data/278841/7 "2021-08-10T10:42:55Z")
This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.