Kibana APM metrics view din't show Opentelemetry JVM metrics

XuQing_Tan · February 18, 2024, 7:37pm

Versions:

ES v8.11.2
opentelemtry agent 2.1.0
jre 17.0.10

I'm setting up the APM dashboard of Keycloak which uses Quarkus, so per the thread Does Elastic APM support Quarkus? - #2 by cyrilleleclerc I used opentelemetry java agent for instrumentation
I can see the latency, throughput, transactions on the Overview tab, but didn't see the JVM metrics on the Metrics tab. (BTW, while using
ES Java APM agent, i only saw the JVM metrics but no Overview data)

XuQing_Tan · February 18, 2024, 7:37pm

attach the screenshot with the latency/throughput/transactions metrics on Overview tab

XuQing_Tan · February 18, 2024, 7:41pm

per the release notes of opentelemetry java agent 2.0.0, there are breaking changes that renamed the metrics Release Version 2.0.0 · open-telemetry/opentelemetry-java-instrumentation · GitHub

however, i tried the older version like opentelemetry java agent 1.32.1, still didn't see the JVM metrics

XuQing_Tan · February 18, 2024, 7:45pm

the keycloak server is deployed in k8s, and i enabled opentelemetry with following env vars

stephenb · February 18, 2024, 11:19pm

Hi @XuQing_Tan Welcome to the community.

1st, please do not post images of text; they are hard to read, can not be debugged, searched, etc, and some people can not see them at all.

So I think I am seeing the same thing, and I have alerted the engineering team.

BUT I went back to 1.32.1 here

And I am seeing metrics with the old field names...

Thanks for Alerting US!
BTW I don't think OTEL collects System Metrics

Here are my setting

export OTEL_RESOURCE_ATTRIBUTES=service.name=cardatabase-local-otel,service.version=1.1,deployment.environment=production,agent.id=hyperion,host.name=hyperion
export OTEL_EXPORTER_OTLP_ENDPOINT=https://myurl.apm.us-west1.gcp.cloud.es.io:443/
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer asdfasdfawdfsOgskmL"
export OTEL_METRIC_EXPORT_INTERVAL=15000
export OTEL_TRACES_EXPORTER=otlp
export OTEL_METRICS_EXPORTER=otlp
export OTEL_LOGS_EXPORTER=otlp

# Command
java -javaagent:./opentelemetry-javaagent-1.32.1.jar  -jar target/cardatabase-0.0.1-SNAPSHOT.jar

XuQing_Tan · February 19, 2024, 12:51am

hi Stephen

thanks for looking into this issue.

I'm using 1.32.1 (see the log)

Appending additional Java properties to JAVA_OPTS: -javaagent:/opentelemetry-javaagent.jar -Xms768m -Xmx1536m
OpenJDK 64-Bit Server VM warning: Sharing is only supported for boot loader classes because bootstrap classpath has been appended
[otel.javaagent 2024-02-18 23:53:28:829 +0000] [main] INFO io.opentelemetry.javaagent.tooling.VersionLogger - opentelemetry-javaagent - version: 1.32.1
The following build time non-cli properties were found, but will be ignored during run time: kc.cache, kc.cache-stack, kc.db, kc.http-relative-path, kc.health-enabled
2024-02-18 23:53:32,434 INFO  [org.keycloak.quarkus.runtime.hostname.DefaultHostnameProvider] (main) Hostname settings: Base URL: https://aaa, Hostname: aaa, Strict HTTPS: true, Path: /iam, Strict BackChannel: false, Admin URL: https://aaa, Admin: aaa, Port: -1, Proxied: true
2024-02-18 23:53:35,774 WARN  [io.quarkus.agroal.runtime.DataSources] (main) Datasource <default> enables XA but transaction recovery is not enabled. Please enable transaction recovery by setting quarkus.transaction-manager.enable-recovery=true, otherwise data may be lost if the application is terminated abruptly
2024-02-18 23:53:37,116 WARN  [org.infinispan.PERSISTENCE] (keycloak-cache-init) ISPN000554: jboss-marshalling is deprecated and planned for removal
2024-02-18 23:53:37,319 INFO  [org.infinispan.CONTAINER] (keycloak-cache-init) ISPN000556: Starting user marshaller 'org.infinispan.jboss.marshalling.core.JBossUserMarshaller'
2024-02-18 23:53:38,150 INFO  [org.infinispan.CLUSTER] (keycloak-cache-init) ISPN000078: Starting JGroups channel `ISPN` with stack `kubernetes`
2024-02-18 23:53:38,152 INFO  [org.jgroups.JChannel] (keycloak-cache-init) local_addr: f043a2b1-0501-4b3d-ba54-5018d1175bbf, name: keycloak-0-26677
2024-02-18 23:53:38,163 INFO  [org.jgroups.protocols.FD_SOCK2] (keycloak-cache-init) server listening on *.57800
2024-02-18 23:53:38,354 INFO  [org.keycloak.broker.provider.AbstractIdentityProviderMapper] (main) Registering class org.keycloak.broker.provider.mappersync.ConfigSyncEventListener
2024-02-18 23:53:40,212 INFO  [org.jgroups.protocols.pbcast.GMS] (keycloak-cache-init) keycloak-0-26677: no members discovered after 2043 ms: creating cluster as coordinator
2024-02-18 23:53:40,224 INFO  [org.infinispan.CLUSTER] (keycloak-cache-init) ISPN000094: Received new cluster view for channel ISPN: [keycloak-0-26677|0] (1) [keycloak-0-26677]
2024-02-18 23:53:40,316 INFO  [org.infinispan.CLUSTER] (keycloak-cache-init) ISPN000079: Channel `ISPN` local address is `keycloak-0-26677`, physical addresses are `[100.126.90.192:7800]`
2024-02-18 23:53:40,331 WARN  [org.infinispan.CONFIG] (keycloak-cache-init) ISPN000569: Unable to persist Infinispan internal caches as no global state enabled
2024-02-18 23:53:40,996 INFO  [org.keycloak.connections.infinispan.DefaultInfinispanConnectionProviderFactory] (main) Node name: keycloak-0-26677, Site name: null\
2024-02-18 23:53:42,522 INFO  [io.quarkus] (main) Keycloak 23.0.6 on JVM (powered by Quarkus 3.2.10.Final) started in 11.809s. Listening on: http://0.0.0.0:8080
2024-02-18 23:53:42,522 INFO  [io.quarkus] (main) Profile prod activated.
2024-02-18 23:53:42,522 INFO  [io.quarkus] (main) Installed features: [agroal, cdi, hibernate-orm, jdbc-h2, jdbc-mariadb, jdbc-mssql, jdbc-mysql, jdbc-oracle, jdbc-postgresql, keycloak, logging-gelf, micrometer, narayana-jta, reactive-routes, resteasy-reactive, resteasy-reactive-jackson, smallrye-context-propagation, smallrye-health, vertx]

but didn't see the metrics

my settings

    env:
    - name: JAVA_OPTS_APPEND
      value: -javaagent:/opentelemetry-javaagent.jar -Xms768m -Xmx1536m
    - name: OTEL_RESOURCE_ATTRIBUTES
      value: service.name=keycloak,service.version=test,deployment.environment=qa
    - name: OTEL_EXPORTER_OTLP_ENDPOINT
      value: https://aaa:443
    - name: OTEL_EXPORTER_OTLP_HEADERS
      value: Authorization=Bearer xxxx
    - name: OTEL_METRICS_EXPORTER
      value: otlp
    - name: OTEL_LOGS_EXPORTER
      value: otlp
    - name: OTEL_TRACES_EXPORTER
      value: otlp
    - name: OTEL_EXPORTER_OTLP_PROTOCOL
      value: http/protobuf

stephenb · February 19, 2024, 12:59am

Did you try to set the interval? Otherwise I think it's very long.

Also, I suppose there could be something different about your OS. I'm not familiar with Quarkis

stephenb · February 19, 2024, 1:00am

Why are you using protobuf?
I did not set that.... Not sure if that makes a difference

XuQing_Tan · February 19, 2024, 1:10am

ah, right, the interval really matters, it worked now, thanks @stephenb

XuQing_Tan · February 19, 2024, 1:12am

with opentelemetry agent 1.3x, the default protocol is grpc, which caused issue on my env:

2024-02-18 23:48:28,565 INFO  [io.quarkus] (main) Keycloak 23.0.6 on JVM (powered by Quarkus 3.2.10.Final) started in 11.803s. Listening on: http://0.0.0.0:8080
2024-02-18 23:48:28,565 INFO  [io.quarkus] (main) Profile prod activated.
2024-02-18 23:48:28,566 INFO  [io.quarkus] (main) Installed features: [agroal, cdi, hibernate-orm, jdbc-h2, jdbc-mariadb, jdbc-mssql, jdbc-mysql, jdbc-oracle, jdbc-postgresql, keycloak, logging-gelf, micrometer, narayana-jta, reactive-routes, resteasy-reactive, resteasy-reactive-jackson, smallrye-context-propagation, smallrye-health, vertx]
[otel.javaagent 2024-02-18 23:48:30:328 +0000] [OkHttp https://aaa/...] WARN io.opentelemetry.exporter.internal.grpc.GrpcExporter - Failed to export spans. Server responded with gRPC status code 2. Error message:
[otel.javaagent 2024-02-18 23:48:40:342 +0000] [OkHttp https://aaa/...] WARN io.opentelemetry.exporter.internal.grpc.GrpcExporter - Failed to export spans. Server responded with gRPC status code 2. Error message:
[otel.javaagent 2024-02-18 23:48:50:355 +0000] [OkHttp https://aaa/...] WARN io.opentelemetry.exporter.internal.grpc.GrpcExporter - Failed to export spans. Server responded with gRPC status code 2. Error message:
[otel.javaagent 2024-02-18 23:49:00:369 +0000] [OkHttp https://aaa/...] WARN io.opentelemetry.exporter.internal.grpc.GrpcExporter - Failed to export spans. Server responded with gRPC status code 2. Error message:
[otel.javaagent 2024-02-18 23:49:10:384 +0000] [OkHttp https://aaa/...] WARN io.opentelemetry.exporter.internal.grpc.GrpcExporter - Failed to export spans. Server responded with gRPC status code 2. Error message:
[otel.javaagent 2024-02-18 23:49:15:307 +0000] [OkHttp https://aaa/...] WARN io.opentelemetry.exporter.internal.grpc.GrpcExporter - Failed to export metrics. Server responded with gRPC status code 2. Error message:
[otel.javaagent 2024-02-18 23:49:20:397 +0000] [OkHttp https://aaa/...] WARN io.opentelemetry.exporter.internal.grpc.GrpcExporter - Failed to export spans. Server responded with gRPC status code 2. Error message:
[otel.javaagent 2024-02-18 23:49:30:410 +0000] [OkHttp https://aaa/...] WARN io.opentelemetry.exporter.internal.grpc.GrpcExporter - Failed to export spans. Server responded with gRPC status code 2. Error message:
[otel.javaagent 2024-02-18 23:49:40:424 +0000] [OkHttp https://aaa/...] WARN io.opentelemetry.exporter.internal.grpc.GrpcExporter - Failed to export spans. Server responded with gRPC status code 2. Error message:
[otel.javaagent 2024-02-18 23:49:50:437 +0000] [OkHttp https://aaa/...] WARN io.opentelemetry.exporter.internal.grpc.GrpcExporter - Failed to export spans. Server responded with gRPC status code 2. Error message:
[otel.javaagent 2024-02-18 23:50:00:451 +0000] [OkHttp https://aaa/...] WARN io.opentelemetry.exporter.internal.grpc.GrpcExporter - Failed to export spans. Server responded with gRPC status code 2. Error message:
[otel.javaagent 2024-02-18 23:50:10:464 +0000] [OkHttp https://aaa/...] WARN io.opentelemetry.exporter.internal.grpc.GrpcExporter - Failed to export spans. Server responded with gRPC status code 2. Error message:
[otel.javaagent 2024-02-18 23:50:15:194 +0000] [OkHttp https://aaa/...] WARN io.opentelemetry.exporter.internal.grpc.GrpcExporter - Failed to export metrics. Server responded with gRPC status code 2. Error message:
[otel.javaagent 2024-02-18 23:50:20:478 +0000] [OkHttp https://aaa/...] WARN io.opentelemetry.exporter.internal.grpc.GrpcExporter - Failed to export spans. Server responded with gRPC status code 2. Error message:
[otel.javaagent 2024-02-18 23:50:25:491 +0000] [OkHttp https://aaa/...] WARN io.opentelemetry.exporter.internal.grpc.GrpcExporter - Failed to export spans. Server responded with gRPC status code 2. Error message:
[otel.javaagent 2024-02-18 23:50:30:504 +0000] [OkHttp https://aaa/...] WARN io.opentelemetry.exporter.internal.grpc.GrpcExporter - Failed to export spans. Server responded with gRPC status code 2. Error message:
[otel.javaagent 2024-02-18 23:50:40:518 +0000] [OkHttp https://aaa/...] WARN io.opentelemetry.exporter.internal.grpc.GrpcExporter - Failed to export spans. Server responded with gRPC status code 2. Error message:
[otel.javaagent 2024-02-18 23:50:50:532 +0000] [OkHttp https://aaa/...] WARN io.opentelemetry.exporter.internal.grpc.GrpcExporter - Failed to export spans. Server responded with gRPC status code 2. Error message:
[otel.javaagent 2024-02-18 23:51:00:545 +0000] [OkHttp https://aaa/...] WARN io.opentelemetry.exporter.internal.grpc.GrpcExporter - Too many log messages detected. Will only log once per minute from now on.

that's why i have to change the protocol to protobuf

stephenb · February 19, 2024, 2:49pm

BTW @XuQing_Tan

Known issue, team is working on some other priorities at the moment.

github.com/elastic/kibana

Update APM dashboards with stable JVM OTel metrics

opened 12:26PM - 08 Jan 24 UTC

SylvainJuge

Team:APM apm:service-metrics

**Describe the feature:** OpenTelemetry JVM metrics are now stable and part o…f the Semantic Conventions, thus we need to align the APM dashboards to use them: - this will be part of the OpenTelemetry 1.28.0 release (not yet created as of writing, see https://github.com/open-telemetry/opentelemetry-specification/releases for updates). - comment mentioning that it will be part of the next release: https://github.com/open-telemetry/opentelemetry-specification/issues/3311#issuecomment-1853050460. - compatibility with previous versions might not be a requirement and could be provided through documentation and custom dashboards. **Describe a specific use case for the feature:**

system · March 18, 2024, 2:49pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.