Versions:
I'm setting up the APM dashboard of Keycloak which uses Quarkus, so per the thread Does Elastic APM support Quarkus? - #2 by cyrilleleclerc I used opentelemetry java agent for instrumentation
I can see the latency, throughput, transactions on the Overview tab, but didn't see the JVM metrics on the Metrics tab. (BTW, while using
ES Java APM agent, i only saw the JVM metrics but no Overview data)
per the release notes of opentelemetry java agent 2.0.0, there are breaking changes that renamed the metrics Release Version 2.0.0 · open-telemetry/opentelemetry-java-instrumentation · GitHub
however, i tried the older version like opentelemetry java agent 1.32.1, still didn't see the JVM metrics
Hi @XuQing_Tan Welcome to the community.
1st, please do not post images of text; they are hard to read, can not be debugged, searched, etc, and some people can not see them at all.
So I think I am seeing the same thing, and I have alerted the engineering team.
BUT I went back to 1.32.1 here
And I am seeing metrics with the old field names...
Thanks for Alerting US!
BTW I don't think OTEL collects System Metrics
Here are my setting
export OTEL_RESOURCE_ATTRIBUTES=service.name=cardatabase-local-otel,service.version=1.1,deployment.environment=production,agent.id=hyperion,host.name=hyperion
export OTEL_EXPORTER_OTLP_ENDPOINT=https://myurl.apm.us-west1.gcp.cloud.es.io:443/
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer asdfasdfawdfsOgskmL"
export OTEL_METRIC_EXPORT_INTERVAL=15000
export OTEL_TRACES_EXPORTER=otlp
export OTEL_METRICS_EXPORTER=otlp
export OTEL_LOGS_EXPORTER=otlp
# Command
java -javaagent:./opentelemetry-javaagent-1.32.1.jar -jar target/cardatabase-0.0.1-SNAPSHOT.jar
hi Stephen
thanks for looking into this issue.
I'm using 1.32.1 (see the log)
Appending additional Java properties to JAVA_OPTS: -javaagent:/opentelemetry-javaagent.jar -Xms768m -Xmx1536m
OpenJDK 64-Bit Server VM warning: Sharing is only supported for boot loader classes because bootstrap classpath has been appended
[otel.javaagent 2024-02-18 23:53:28:829 +0000] [main] INFO io.opentelemetry.javaagent.tooling.VersionLogger - opentelemetry-javaagent - version: 1.32.1
The following build time non-cli properties were found, but will be ignored during run time: kc.cache, kc.cache-stack, kc.db, kc.http-relative-path, kc.health-enabled
2024-02-18 23:53:32,434 INFO [org.keycloak.quarkus.runtime.hostname.DefaultHostnameProvider] (main) Hostname settings: Base URL: https://aaa, Hostname: aaa, Strict HTTPS: true, Path: /iam, Strict BackChannel: false, Admin URL: https://aaa, Admin: aaa, Port: -1, Proxied: true
2024-02-18 23:53:35,774 WARN [io.quarkus.agroal.runtime.DataSources] (main) Datasource <default> enables XA but transaction recovery is not enabled. Please enable transaction recovery by setting quarkus.transaction-manager.enable-recovery=true, otherwise data may be lost if the application is terminated abruptly
2024-02-18 23:53:37,116 WARN [org.infinispan.PERSISTENCE] (keycloak-cache-init) ISPN000554: jboss-marshalling is deprecated and planned for removal
2024-02-18 23:53:37,319 INFO [org.infinispan.CONTAINER] (keycloak-cache-init) ISPN000556: Starting user marshaller 'org.infinispan.jboss.marshalling.core.JBossUserMarshaller'
2024-02-18 23:53:38,150 INFO [org.infinispan.CLUSTER] (keycloak-cache-init) ISPN000078: Starting JGroups channel `ISPN` with stack `kubernetes`
2024-02-18 23:53:38,152 INFO [org.jgroups.JChannel] (keycloak-cache-init) local_addr: f043a2b1-0501-4b3d-ba54-5018d1175bbf, name: keycloak-0-26677
2024-02-18 23:53:38,163 INFO [org.jgroups.protocols.FD_SOCK2] (keycloak-cache-init) server listening on *.57800
2024-02-18 23:53:38,354 INFO [org.keycloak.broker.provider.AbstractIdentityProviderMapper] (main) Registering class org.keycloak.broker.provider.mappersync.ConfigSyncEventListener
2024-02-18 23:53:40,212 INFO [org.jgroups.protocols.pbcast.GMS] (keycloak-cache-init) keycloak-0-26677: no members discovered after 2043 ms: creating cluster as coordinator
2024-02-18 23:53:40,224 INFO [org.infinispan.CLUSTER] (keycloak-cache-init) ISPN000094: Received new cluster view for channel ISPN: [keycloak-0-26677|0] (1) [keycloak-0-26677]
2024-02-18 23:53:40,316 INFO [org.infinispan.CLUSTER] (keycloak-cache-init) ISPN000079: Channel `ISPN` local address is `keycloak-0-26677`, physical addresses are `[100.126.90.192:7800]`
2024-02-18 23:53:40,331 WARN [org.infinispan.CONFIG] (keycloak-cache-init) ISPN000569: Unable to persist Infinispan internal caches as no global state enabled
2024-02-18 23:53:40,996 INFO [org.keycloak.connections.infinispan.DefaultInfinispanConnectionProviderFactory] (main) Node name: keycloak-0-26677, Site name: null\
2024-02-18 23:53:42,522 INFO [io.quarkus] (main) Keycloak 23.0.6 on JVM (powered by Quarkus 3.2.10.Final) started in 11.809s. Listening on: http://0.0.0.0:8080
2024-02-18 23:53:42,522 INFO [io.quarkus] (main) Profile prod activated.
2024-02-18 23:53:42,522 INFO [io.quarkus] (main) Installed features: [agroal, cdi, hibernate-orm, jdbc-h2, jdbc-mariadb, jdbc-mssql, jdbc-mysql, jdbc-oracle, jdbc-postgresql, keycloak, logging-gelf, micrometer, narayana-jta, reactive-routes, resteasy-reactive, resteasy-reactive-jackson, smallrye-context-propagation, smallrye-health, vertx]
but didn't see the metrics
my settings
env:
- name: JAVA_OPTS_APPEND
value: -javaagent:/opentelemetry-javaagent.jar -Xms768m -Xmx1536m
- name: OTEL_RESOURCE_ATTRIBUTES
value: service.name=keycloak,service.version=test,deployment.environment=qa
- name: OTEL_EXPORTER_OTLP_ENDPOINT
value: https://aaa:443
- name: OTEL_EXPORTER_OTLP_HEADERS
value: Authorization=Bearer xxxx
- name: OTEL_METRICS_EXPORTER
value: otlp
- name: OTEL_LOGS_EXPORTER
value: otlp
- name: OTEL_TRACES_EXPORTER
value: otlp
- name: OTEL_EXPORTER_OTLP_PROTOCOL
value: http/protobuf
Did you try to set the interval? Otherwise I think it's very long.
Also, I suppose there could be something different about your OS. I'm not familiar with Quarkis
Why are you using protobuf?
I did not set that.... Not sure if that makes a difference
ah, right, the interval really matters, it worked now, thanks @stephenb
with opentelemetry agent 1.3x, the default protocol is grpc, which caused issue on my env:
2024-02-18 23:48:28,565 INFO [io.quarkus] (main) Keycloak 23.0.6 on JVM (powered by Quarkus 3.2.10.Final) started in 11.803s. Listening on: http://0.0.0.0:8080
2024-02-18 23:48:28,565 INFO [io.quarkus] (main) Profile prod activated.
2024-02-18 23:48:28,566 INFO [io.quarkus] (main) Installed features: [agroal, cdi, hibernate-orm, jdbc-h2, jdbc-mariadb, jdbc-mssql, jdbc-mysql, jdbc-oracle, jdbc-postgresql, keycloak, logging-gelf, micrometer, narayana-jta, reactive-routes, resteasy-reactive, resteasy-reactive-jackson, smallrye-context-propagation, smallrye-health, vertx]
[otel.javaagent 2024-02-18 23:48:30:328 +0000] [OkHttp https://aaa/...] WARN io.opentelemetry.exporter.internal.grpc.GrpcExporter - Failed to export spans. Server responded with gRPC status code 2. Error message:
[otel.javaagent 2024-02-18 23:48:40:342 +0000] [OkHttp https://aaa/...] WARN io.opentelemetry.exporter.internal.grpc.GrpcExporter - Failed to export spans. Server responded with gRPC status code 2. Error message:
[otel.javaagent 2024-02-18 23:48:50:355 +0000] [OkHttp https://aaa/...] WARN io.opentelemetry.exporter.internal.grpc.GrpcExporter - Failed to export spans. Server responded with gRPC status code 2. Error message:
[otel.javaagent 2024-02-18 23:49:00:369 +0000] [OkHttp https://aaa/...] WARN io.opentelemetry.exporter.internal.grpc.GrpcExporter - Failed to export spans. Server responded with gRPC status code 2. Error message:
[otel.javaagent 2024-02-18 23:49:10:384 +0000] [OkHttp https://aaa/...] WARN io.opentelemetry.exporter.internal.grpc.GrpcExporter - Failed to export spans. Server responded with gRPC status code 2. Error message:
[otel.javaagent 2024-02-18 23:49:15:307 +0000] [OkHttp https://aaa/...] WARN io.opentelemetry.exporter.internal.grpc.GrpcExporter - Failed to export metrics. Server responded with gRPC status code 2. Error message:
[otel.javaagent 2024-02-18 23:49:20:397 +0000] [OkHttp https://aaa/...] WARN io.opentelemetry.exporter.internal.grpc.GrpcExporter - Failed to export spans. Server responded with gRPC status code 2. Error message:
[otel.javaagent 2024-02-18 23:49:30:410 +0000] [OkHttp https://aaa/...] WARN io.opentelemetry.exporter.internal.grpc.GrpcExporter - Failed to export spans. Server responded with gRPC status code 2. Error message:
[otel.javaagent 2024-02-18 23:49:40:424 +0000] [OkHttp https://aaa/...] WARN io.opentelemetry.exporter.internal.grpc.GrpcExporter - Failed to export spans. Server responded with gRPC status code 2. Error message:
[otel.javaagent 2024-02-18 23:49:50:437 +0000] [OkHttp https://aaa/...] WARN io.opentelemetry.exporter.internal.grpc.GrpcExporter - Failed to export spans. Server responded with gRPC status code 2. Error message:
[otel.javaagent 2024-02-18 23:50:00:451 +0000] [OkHttp https://aaa/...] WARN io.opentelemetry.exporter.internal.grpc.GrpcExporter - Failed to export spans. Server responded with gRPC status code 2. Error message:
[otel.javaagent 2024-02-18 23:50:10:464 +0000] [OkHttp https://aaa/...] WARN io.opentelemetry.exporter.internal.grpc.GrpcExporter - Failed to export spans. Server responded with gRPC status code 2. Error message:
[otel.javaagent 2024-02-18 23:50:15:194 +0000] [OkHttp https://aaa/...] WARN io.opentelemetry.exporter.internal.grpc.GrpcExporter - Failed to export metrics. Server responded with gRPC status code 2. Error message:
[otel.javaagent 2024-02-18 23:50:20:478 +0000] [OkHttp https://aaa/...] WARN io.opentelemetry.exporter.internal.grpc.GrpcExporter - Failed to export spans. Server responded with gRPC status code 2. Error message:
[otel.javaagent 2024-02-18 23:50:25:491 +0000] [OkHttp https://aaa/...] WARN io.opentelemetry.exporter.internal.grpc.GrpcExporter - Failed to export spans. Server responded with gRPC status code 2. Error message:
[otel.javaagent 2024-02-18 23:50:30:504 +0000] [OkHttp https://aaa/...] WARN io.opentelemetry.exporter.internal.grpc.GrpcExporter - Failed to export spans. Server responded with gRPC status code 2. Error message:
[otel.javaagent 2024-02-18 23:50:40:518 +0000] [OkHttp https://aaa/...] WARN io.opentelemetry.exporter.internal.grpc.GrpcExporter - Failed to export spans. Server responded with gRPC status code 2. Error message:
[otel.javaagent 2024-02-18 23:50:50:532 +0000] [OkHttp https://aaa/...] WARN io.opentelemetry.exporter.internal.grpc.GrpcExporter - Failed to export spans. Server responded with gRPC status code 2. Error message:
[otel.javaagent 2024-02-18 23:51:00:545 +0000] [OkHttp https://aaa/...] WARN io.opentelemetry.exporter.internal.grpc.GrpcExporter - Too many log messages detected. Will only log once per minute from now on.
that's why i have to change the protocol to protobuf
BTW @XuQing_Tan
Known issue, team is working on some other priorities at the moment.
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
© 2020. All Rights Reserved - Elasticsearch
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant logo are trademarks of the Apache Software Foundation in the United States and/or other countries.