# Kibana APM metrics view din't show Opentelemetry JVM metrics
**URL:** https://discuss.elastic.co/t/kibana-apm-metrics-view-dint-show-opentelemetry-jvm-metrics/353524
**Category:** APM
**Tags:** java, open-telemetry
**Created:** [February 18, 2024, 7:37pm UTC](https://discuss.elastic.co/t/kibana-apm-metrics-view-dint-show-opentelemetry-jvm-metrics/353524 "2024-02-18T19:37:14Z")
**Posts on this page:** 12
**Page:** 1
### Author:  [@XuQing\_Tan](https://discuss.elastic.co/u/XuQing_Tan)
#### Post date: [February 18, 2024, 7:37pm UTC](https://discuss.elastic.co/t/kibana-apm-metrics-view-dint-show-opentelemetry-jvm-metrics/353524/1 "2024-02-18T19:37:14Z")
Versions:
- ES v8.11.2
- opentelemtry agent 2.1.0
- jre 17.0.10
I'm setting up the APM dashboard of Keycloak which uses Quarkus, so per the thread [Does Elastic APM support Quarkus? - #2 by cyrilleleclerc](https://discuss.elastic.co/t/does-elastic-apm-support-quarkus/284167/2) I used opentelemetry java agent for instrumentation
I can see the latency, throughput, transactions on the Overview tab, but didn't see the JVM metrics on the Metrics tab. (BTW, while using
ES Java APM agent, i only saw the JVM metrics but no Overview data)
---
### Author:  [@XuQing\_Tan](https://discuss.elastic.co/u/XuQing_Tan)
#### Post date: [February 18, 2024, 7:37pm UTC](https://discuss.elastic.co/t/kibana-apm-metrics-view-dint-show-opentelemetry-jvm-metrics/353524/2 "2024-02-18T19:37:57Z")
attach the screenshot with the latency/throughput/transactions metrics on Overview tab
---
### Author:  [@XuQing\_Tan](https://discuss.elastic.co/u/XuQing_Tan)
#### Post date: [February 18, 2024, 7:41pm UTC](https://discuss.elastic.co/t/kibana-apm-metrics-view-dint-show-opentelemetry-jvm-metrics/353524/3 "2024-02-18T19:41:13Z")
per the release notes of opentelemetry java agent 2.0.0, there are breaking changes that renamed the metrics [Release Version 2.0.0 · open-telemetry/opentelemetry-java-instrumentation · GitHub](https://github.com/open-telemetry/opentelemetry-java-instrumentation/releases/tag/v2.0.0)
however, i tried the older version like opentelemetry java agent 1.32.1, still didn't see the JVM metrics
---
### Author:  [@XuQing\_Tan](https://discuss.elastic.co/u/XuQing_Tan)
#### Post date: [February 18, 2024, 7:45pm UTC](https://discuss.elastic.co/t/kibana-apm-metrics-view-dint-show-opentelemetry-jvm-metrics/353524/4 "2024-02-18T19:45:53Z")
the keycloak server is deployed in k8s, and i enabled opentelemetry with following env vars
---
### Author:  [@stephenb](https://discuss.elastic.co/u/stephenb)
#### Post date: [February 18, 2024, 11:19pm UTC](https://discuss.elastic.co/t/kibana-apm-metrics-view-dint-show-opentelemetry-jvm-metrics/353524/5 "2024-02-18T23:19:33Z")
Hi @XuQing_Tan Welcome to the community.
1st, please do not post images of text; they are hard to read, can not be debugged, searched, etc, and some people can not see them at all.
So I think I am seeing the same thing, and I have alerted the engineering team.
BUT I went back to 1.32.1 [here](https://github.com/open-telemetry/opentelemetry-java-instrumentation/releases/tag/v1.32.1)
And I am seeing metrics with the old field names...
Thanks for Alerting US!
BTW I don't think OTEL collects System Metrics
Here are my setting
```auto
export OTEL_RESOURCE_ATTRIBUTES=service.name=cardatabase-local-otel,service.version=1.1,deployment.environment=production,agent.id=hyperion,host.name=hyperion
export OTEL_EXPORTER_OTLP_ENDPOINT=https://myurl.apm.us-west1.gcp.cloud.es.io:443/
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer asdfasdfawdfsOgskmL"
export OTEL_METRIC_EXPORT_INTERVAL=15000
export OTEL_TRACES_EXPORTER=otlp
export OTEL_METRICS_EXPORTER=otlp
export OTEL_LOGS_EXPORTER=otlp
# Command
java -javaagent:./opentelemetry-javaagent-1.32.1.jar -jar target/cardatabase-0.0.1-SNAPSHOT.jar
```
---
### Author:  [@XuQing\_Tan](https://discuss.elastic.co/u/XuQing_Tan)
#### Post date: [February 19, 2024, 12:51am UTC](https://discuss.elastic.co/t/kibana-apm-metrics-view-dint-show-opentelemetry-jvm-metrics/353524/6 "2024-02-19T00:51:21Z")
> [@stephenb](#):
>
> BUT I went back to 1.32.1 [here](https://github.com/open-telemetry/opentelemetry-java-instrumentation/releases/tag/v1.32.1)
>
> And I am seeing metrics with the old field names...
hi Stephen
thanks for looking into this issue.
I'm using 1.32.1 (see the log)
```auto
Appending additional Java properties to JAVA_OPTS: -javaagent:/opentelemetry-javaagent.jar -Xms768m -Xmx1536m
OpenJDK 64-Bit Server VM warning: Sharing is only supported for boot loader classes because bootstrap classpath has been appended
[otel.javaagent 2024-02-18 23:53:28:829 +0000] [main] INFO io.opentelemetry.javaagent.tooling.VersionLogger - opentelemetry-javaagent - version: 1.32.1
The following build time non-cli properties were found, but will be ignored during run time: kc.cache, kc.cache-stack, kc.db, kc.http-relative-path, kc.health-enabled
2024-02-18 23:53:32,434 INFO [org.keycloak.quarkus.runtime.hostname.DefaultHostnameProvider] (main) Hostname settings: Base URL: https://aaa, Hostname: aaa, Strict HTTPS: true, Path: /iam, Strict BackChannel: false, Admin URL: https://aaa, Admin: aaa, Port: -1, Proxied: true
2024-02-18 23:53:35,774 WARN [io.quarkus.agroal.runtime.DataSources] (main) Datasource enables XA but transaction recovery is not enabled. Please enable transaction recovery by setting quarkus.transaction-manager.enable-recovery=true, otherwise data may be lost if the application is terminated abruptly
2024-02-18 23:53:37,116 WARN [org.infinispan.PERSISTENCE] (keycloak-cache-init) ISPN000554: jboss-marshalling is deprecated and planned for removal
2024-02-18 23:53:37,319 INFO [org.infinispan.CONTAINER] (keycloak-cache-init) ISPN000556: Starting user marshaller 'org.infinispan.jboss.marshalling.core.JBossUserMarshaller'
2024-02-18 23:53:38,150 INFO [org.infinispan.CLUSTER] (keycloak-cache-init) ISPN000078: Starting JGroups channel `ISPN` with stack `kubernetes`
2024-02-18 23:53:38,152 INFO [org.jgroups.JChannel] (keycloak-cache-init) local_addr: f043a2b1-0501-4b3d-ba54-5018d1175bbf, name: keycloak-0-26677
2024-02-18 23:53:38,163 INFO [org.jgroups.protocols.FD_SOCK2] (keycloak-cache-init) server listening on *.57800
2024-02-18 23:53:38,354 INFO [org.keycloak.broker.provider.AbstractIdentityProviderMapper] (main) Registering class org.keycloak.broker.provider.mappersync.ConfigSyncEventListener
2024-02-18 23:53:40,212 INFO [org.jgroups.protocols.pbcast.GMS] (keycloak-cache-init) keycloak-0-26677: no members discovered after 2043 ms: creating cluster as coordinator
2024-02-18 23:53:40,224 INFO [org.infinispan.CLUSTER] (keycloak-cache-init) ISPN000094: Received new cluster view for channel ISPN: [keycloak-0-26677|0] (1) [keycloak-0-26677]
2024-02-18 23:53:40,316 INFO [org.infinispan.CLUSTER] (keycloak-cache-init) ISPN000079: Channel `ISPN` local address is `keycloak-0-26677`, physical addresses are `[100.126.90.192:7800]`
2024-02-18 23:53:40,331 WARN [org.infinispan.CONFIG] (keycloak-cache-init) ISPN000569: Unable to persist Infinispan internal caches as no global state enabled
2024-02-18 23:53:40,996 INFO [org.keycloak.connections.infinispan.DefaultInfinispanConnectionProviderFactory] (main) Node name: keycloak-0-26677, Site name: null\
2024-02-18 23:53:42,522 INFO [io.quarkus] (main) Keycloak 23.0.6 on JVM (powered by Quarkus 3.2.10.Final) started in 11.809s. Listening on: http://0.0.0.0:8080
2024-02-18 23:53:42,522 INFO [io.quarkus] (main) Profile prod activated.
2024-02-18 23:53:42,522 INFO [io.quarkus] (main) Installed features: [agroal, cdi, hibernate-orm, jdbc-h2, jdbc-mariadb, jdbc-mssql, jdbc-mysql, jdbc-oracle, jdbc-postgresql, keycloak, logging-gelf, micrometer, narayana-jta, reactive-routes, resteasy-reactive, resteasy-reactive-jackson, smallrye-context-propagation, smallrye-health, vertx]
```
but didn't see the metrics
my settings
```auto
env:
- name: JAVA_OPTS_APPEND
value: -javaagent:/opentelemetry-javaagent.jar -Xms768m -Xmx1536m
- name: OTEL_RESOURCE_ATTRIBUTES
value: service.name=keycloak,service.version=test,deployment.environment=qa
- name: OTEL_EXPORTER_OTLP_ENDPOINT
value: https://aaa:443
- name: OTEL_EXPORTER_OTLP_HEADERS
value: Authorization=Bearer xxxx
- name: OTEL_METRICS_EXPORTER
value: otlp
- name: OTEL_LOGS_EXPORTER
value: otlp
- name: OTEL_TRACES_EXPORTER
value: otlp
- name: OTEL_EXPORTER_OTLP_PROTOCOL
value: http/protobuf
```
---
### Author:  [@stephenb](https://discuss.elastic.co/u/stephenb)
#### Post date: [February 19, 2024, 12:59am UTC](https://discuss.elastic.co/t/kibana-apm-metrics-view-dint-show-opentelemetry-jvm-metrics/353524/7 "2024-02-19T00:59:33Z")
> [@stephenb](#):
>
> `OTEL_METRIC_EXPORT_INTERVAL=15000`
Did you try to set the interval? Otherwise I think it's very long.
Also, I suppose there could be something different about your OS. I'm not familiar with Quarkis
---
### Author:  [@stephenb](https://discuss.elastic.co/u/stephenb)
#### Post date: [February 19, 2024, 1:00am UTC](https://discuss.elastic.co/t/kibana-apm-metrics-view-dint-show-opentelemetry-jvm-metrics/353524/8 "2024-02-19T01:00:09Z")
> [@XuQing\_Tan](#):
>
> `OTEL_EXPORTER_OTLP_PROTOCOL`
Why are you using protobuf?
I did not set that.... Not sure if that makes a difference
---
### Author:  [@XuQing\_Tan](https://discuss.elastic.co/u/XuQing_Tan)
#### Post date: [February 19, 2024, 1:10am UTC](https://discuss.elastic.co/t/kibana-apm-metrics-view-dint-show-opentelemetry-jvm-metrics/353524/9 "2024-02-19T01:10:27Z")
ah, right, the interval really matters, it worked now, thanks @stephenb
---
### Author:  [@XuQing\_Tan](https://discuss.elastic.co/u/XuQing_Tan)
#### Post date: [February 19, 2024, 1:12am UTC](https://discuss.elastic.co/t/kibana-apm-metrics-view-dint-show-opentelemetry-jvm-metrics/353524/10 "2024-02-19T01:12:19Z")
with opentelemetry agent 1.3x, the default protocol is grpc, which caused issue on my env:
```auto
2024-02-18 23:48:28,565 INFO [io.quarkus] (main) Keycloak 23.0.6 on JVM (powered by Quarkus 3.2.10.Final) started in 11.803s. Listening on: http://0.0.0.0:8080
2024-02-18 23:48:28,565 INFO [io.quarkus] (main) Profile prod activated.
2024-02-18 23:48:28,566 INFO [io.quarkus] (main) Installed features: [agroal, cdi, hibernate-orm, jdbc-h2, jdbc-mariadb, jdbc-mssql, jdbc-mysql, jdbc-oracle, jdbc-postgresql, keycloak, logging-gelf, micrometer, narayana-jta, reactive-routes, resteasy-reactive, resteasy-reactive-jackson, smallrye-context-propagation, smallrye-health, vertx]
[otel.javaagent 2024-02-18 23:48:30:328 +0000] [OkHttp https://aaa/...] WARN io.opentelemetry.exporter.internal.grpc.GrpcExporter - Failed to export spans. Server responded with gRPC status code 2. Error message:
[otel.javaagent 2024-02-18 23:48:40:342 +0000] [OkHttp https://aaa/...] WARN io.opentelemetry.exporter.internal.grpc.GrpcExporter - Failed to export spans. Server responded with gRPC status code 2. Error message:
[otel.javaagent 2024-02-18 23:48:50:355 +0000] [OkHttp https://aaa/...] WARN io.opentelemetry.exporter.internal.grpc.GrpcExporter - Failed to export spans. Server responded with gRPC status code 2. Error message:
[otel.javaagent 2024-02-18 23:49:00:369 +0000] [OkHttp https://aaa/...] WARN io.opentelemetry.exporter.internal.grpc.GrpcExporter - Failed to export spans. Server responded with gRPC status code 2. Error message:
[otel.javaagent 2024-02-18 23:49:10:384 +0000] [OkHttp https://aaa/...] WARN io.opentelemetry.exporter.internal.grpc.GrpcExporter - Failed to export spans. Server responded with gRPC status code 2. Error message:
[otel.javaagent 2024-02-18 23:49:15:307 +0000] [OkHttp https://aaa/...] WARN io.opentelemetry.exporter.internal.grpc.GrpcExporter - Failed to export metrics. Server responded with gRPC status code 2. Error message:
[otel.javaagent 2024-02-18 23:49:20:397 +0000] [OkHttp https://aaa/...] WARN io.opentelemetry.exporter.internal.grpc.GrpcExporter - Failed to export spans. Server responded with gRPC status code 2. Error message:
[otel.javaagent 2024-02-18 23:49:30:410 +0000] [OkHttp https://aaa/...] WARN io.opentelemetry.exporter.internal.grpc.GrpcExporter - Failed to export spans. Server responded with gRPC status code 2. Error message:
[otel.javaagent 2024-02-18 23:49:40:424 +0000] [OkHttp https://aaa/...] WARN io.opentelemetry.exporter.internal.grpc.GrpcExporter - Failed to export spans. Server responded with gRPC status code 2. Error message:
[otel.javaagent 2024-02-18 23:49:50:437 +0000] [OkHttp https://aaa/...] WARN io.opentelemetry.exporter.internal.grpc.GrpcExporter - Failed to export spans. Server responded with gRPC status code 2. Error message:
[otel.javaagent 2024-02-18 23:50:00:451 +0000] [OkHttp https://aaa/...] WARN io.opentelemetry.exporter.internal.grpc.GrpcExporter - Failed to export spans. Server responded with gRPC status code 2. Error message:
[otel.javaagent 2024-02-18 23:50:10:464 +0000] [OkHttp https://aaa/...] WARN io.opentelemetry.exporter.internal.grpc.GrpcExporter - Failed to export spans. Server responded with gRPC status code 2. Error message:
[otel.javaagent 2024-02-18 23:50:15:194 +0000] [OkHttp https://aaa/...] WARN io.opentelemetry.exporter.internal.grpc.GrpcExporter - Failed to export metrics. Server responded with gRPC status code 2. Error message:
[otel.javaagent 2024-02-18 23:50:20:478 +0000] [OkHttp https://aaa/...] WARN io.opentelemetry.exporter.internal.grpc.GrpcExporter - Failed to export spans. Server responded with gRPC status code 2. Error message:
[otel.javaagent 2024-02-18 23:50:25:491 +0000] [OkHttp https://aaa/...] WARN io.opentelemetry.exporter.internal.grpc.GrpcExporter - Failed to export spans. Server responded with gRPC status code 2. Error message:
[otel.javaagent 2024-02-18 23:50:30:504 +0000] [OkHttp https://aaa/...] WARN io.opentelemetry.exporter.internal.grpc.GrpcExporter - Failed to export spans. Server responded with gRPC status code 2. Error message:
[otel.javaagent 2024-02-18 23:50:40:518 +0000] [OkHttp https://aaa/...] WARN io.opentelemetry.exporter.internal.grpc.GrpcExporter - Failed to export spans. Server responded with gRPC status code 2. Error message:
[otel.javaagent 2024-02-18 23:50:50:532 +0000] [OkHttp https://aaa/...] WARN io.opentelemetry.exporter.internal.grpc.GrpcExporter - Failed to export spans. Server responded with gRPC status code 2. Error message:
[otel.javaagent 2024-02-18 23:51:00:545 +0000] [OkHttp https://aaa/...] WARN io.opentelemetry.exporter.internal.grpc.GrpcExporter - Too many log messages detected. Will only log once per minute from now on.
```
that's why i have to change the protocol to protobuf
---
### Author:  [@stephenb](https://discuss.elastic.co/u/stephenb)
#### Post date: [February 19, 2024, 2:49pm UTC](https://discuss.elastic.co/t/kibana-apm-metrics-view-dint-show-opentelemetry-jvm-metrics/353524/11 "2024-02-19T14:49:29Z")
BTW @XuQing_Tan
Known issue, team is working on some other priorities at the moment.
>
>
> \*\*Describe the feature:\*\*
>
> OpenTelemetry JVM metrics are now stable and part o…f the Semantic Conventions, thus we need to align the APM dashboards to use them:
> \- this will be part of the OpenTelemetry 1.28.0 release (not yet created as of writing, see
> https://github.com/open-telemetry/opentelemetry-specification/releases for updates).
> \- comment mentioning that it will be part of the next release: https://github.com/open-telemetry/opentelemetry-specification/issues/3311#issuecomment-1853050460.
> \- compatibility with previous versions might not be a requirement and could be provided through documentation and custom dashboards.
>
> \*\*Describe a specific use case for the feature:\*\*
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [March 18, 2024, 2:49pm UTC](https://discuss.elastic.co/t/kibana-apm-metrics-view-dint-show-opentelemetry-jvm-metrics/353524/12 "2024-03-18T14:49:50Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.