No JVMs were found APM

User28 · November 4, 2022, 10:16am

Hi,
Elasticsearch - v7.17.4
Kibana - v7.17.4
APM Server - v7.17.4
Opentelemetry agent - v1.11.0

In Kibana APM JVM UI, it says No JVMs were found

service.node.name is also set

We have sending observability data from Opentelemetry collector to Elastic APM. We see all timelines, latency, logs, errors but not not JVM metrics

User28 · November 6, 2022, 5:18pm

Is this the limitation?

stephenb · November 6, 2022, 5:34pm

Perhaps this?

And / Or are you adding the correct modules to collect the JVM Metrics?

opentelemetry-java-contrib: Provides helpful libraries and standalone OpenTelemetry-based utilities that don’t fit the express scope of the OpenTelemetry Java or Java Instrumentation projects. For example, JMX metric gathering.

I am running 8.4.3 and do not see JVM but I have not added the contrib JMX so I am a little confused as well... I will try 8.5.0 and see what I see.... Nope I do not see in 8.5.0 as well...

Perhaps @alexander.wert can help both of us

Do we need something besides just the base Java OTEL Agent? here

Do we need to run the jmx contrib as a separate process?

stephenb · November 6, 2022, 7:25pm

Just for reference my start up looks like this .. I see traces transaction call to DB but no JVMs

export OTEL_RESOURCE_ATTRIBUTES=service.name=cardatabase-local-otel,service.version=1.1,deployment.environment=production
export OTEL_EXPORTER_OTLP_ENDPOINT=https://adsasfdsafd.apm.us-west2.gcp.elastic-cloud.com
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer asdfasdfasdfasdfasdf"
export OTEL_METRICS_EXPORTER="otlp"
export OTEL_LOGS_EXPORTER="otlp"
java -javaagent:./opentelemetry-javaagent.jar \
     -jar target/cardatabase-0.0.1-SNAPSHOT.jar

stephenb · November 6, 2022, 7:56pm

I tried using the contrib, still confused whether I need this or not the Missing JVM Github issue above seems to lead me to think not.. but without contrib I get nothing.

Note this is on Elastic Stack 8.5.0

Now App Startup

export OTEL_RESOURCE_ATTRIBUTES=service.name=cardatabase-local-otel,service.version=1.1,deployment.environment=production
export OTEL_EXPORTER_OTLP_ENDPOINT=https://test.apm.us-west2.gcp.elastic-cloud.com
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer fb6mQwEYrKe3KC4tiO"
export OTEL_METRICS_EXPORTER="otlp"
export OTEL_LOGS_EXPORTER="otlp"
java -javaagent:./opentelemetry-javaagent.jar \
    -Dcom.sun.management.jmxremote \
    -Dcom.sun.management.jmxremote.port=9010 \
    -Dcom.sun.management.jmxremote.local.only=false \
    -Dcom.sun.management.jmxremote.authenticate=false \
    -Dcom.sun.management.jmxremote.ssl=false \
    -Djava.rmi.server.hostname=localhost \
    -jar target/cardatabase-0.0.1-SNAPSHOT.jar

With contrib...

session.properties

otel.jmx.service.url = service:jmx:rmi:///jndi/rmi://localhost:9010/jmxrmi
otel.jmx.target.system = jvm
otel.jmx.interval.milliseconds = 5000
otel.metrics.exporter = otlp
otel.exporter.otlp.endpoint = https://sdgfsddsfg.apm.us-west2.gcp.elastic-cloud.com
otel.exporter.otlp.headers = "Authorization=Bearer sfgsdfgsdfg"

Run it .. get a few errors... then looks like it is shipping

 java -jar opentelemetry-jmx-metrics.jar -config ./session.properties 
Nov 06, 2022 11:54:35 AM io.opentelemetry.contrib.jmxmetrics.JmxMetrics start
INFO: Started GroovyRunner.
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by io.opentelemetry.contrib.jmxmetrics.JmxClient (file:/Users/sbrown/workspace/repos/bvader/tmp/cardatabase/opentelemetry-jmx-metrics.jar) to constructor com.sun.security.sasl.Provider()
WARNING: Please consider reporting this to the maintainers of io.opentelemetry.contrib.jmxmetrics.JmxClient
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
Nov 06, 2022 11:54:44 AM io.opentelemetry.sdk.internal.ThrottlingLogger doLog
WARNING: Instrument jvm.classes.loaded has recorded multiple values for the same attributes.
Nov 06, 2022 11:54:44 AM io.opentelemetry.sdk.internal.ThrottlingLogger doLog
WARNING: Instrument jvm.gc.collections.count has recorded multiple values for the same attributes.
Nov 06, 2022 11:54:44 AM io.opentelemetry.sdk.internal.ThrottlingLogger doLog
WARNING: Instrument jvm.gc.collections.count has recorded multiple values for the same attributes.
Nov 06, 2022 11:54:44 AM io.opentelemetry.sdk.internal.ThrottlingLogger doLog
WARNING: Instrument jvm.gc.collections.elapsed has recorded multiple values for the same attributes.
Nov 06, 2022 11:54:44 AM io.opentelemetry.sdk.internal.ThrottlingLogger doLog
WARNING: Instrument jvm.gc.collections.elapsed has recorded multiple values for the same attributes.
Nov 06, 2022 11:54:44 AM io.opentelemetry.sdk.internal.ThrottlingLogger doLog
WARNING: Instrument jvm.memory.heap.committed has recorded multiple values for the same attributes.
Nov 06, 2022 11:54:44 AM io.opentelemetry.sdk.internal.ThrottlingLogger doLog
WARNING: Instrument jvm.memory.heap.init has recorded multiple values for the same attributes.
Nov 06, 2022 11:54:44 AM io.opentelemetry.sdk.internal.ThrottlingLogger doLog
WARNING: Instrument jvm.memory.heap.max has recorded multiple values for the same attributes.
Nov 06, 2022 11:54:44 AM io.opentelemetry.sdk.internal.ThrottlingLogger doLog
WARNING: Instrument jvm.memory.heap.used has recorded multiple values for the same attributes.
Nov 06, 2022 11:54:44 AM io.opentelemetry.sdk.internal.ThrottlingLogger doLog
WARNING: Instrument jvm.memory.nonheap.committed has recorded multiple values for the same attributes.
Nov 06, 2022 11:54:44 AM io.opentelemetry.sdk.internal.ThrottlingLogger doLog
WARNING: Instrument jvm.memory.nonheap.init has recorded multiple values for the same attributes.
Nov 06, 2022 11:54:44 AM io.opentelemetry.sdk.internal.ThrottlingLogger doLog
WARNING: Instrument jvm.memory.nonheap.max has recorded multiple values for the same attributes.
Nov 06, 2022 11:54:44 AM io.opentelemetry.sdk.internal.ThrottlingLogger doLog
WARNING: Instrument jvm.memory.nonheap.used has recorded multiple values for the same attributes.
Nov 06, 2022 11:54:44 AM io.opentelemetry.sdk.internal.ThrottlingLogger doLog
WARNING: Instrument jvm.memory.pool.committed has recorded multiple values for the same attributes.
Nov 06, 2022 11:54:44 AM io.opentelemetry.sdk.internal.ThrottlingLogger doLog
WARNING: Instrument jvm.memory.pool.committed has recorded multiple values for the same attributes.
Nov 06, 2022 11:54:44 AM io.opentelemetry.sdk.internal.ThrottlingLogger doLog
WARNING: Instrument jvm.memory.pool.committed has recorded multiple values for the same attributes.
Nov 06, 2022 11:54:44 AM io.opentelemetry.sdk.internal.ThrottlingLogger doLog
WARNING: Instrument jvm.memory.pool.committed has recorded multiple values for the same attributes.
Nov 06, 2022 11:54:44 AM io.opentelemetry.sdk.internal.ThrottlingLogger doLog
WARNING: Instrument jvm.memory.pool.committed has recorded multiple values for the same attributes.
Nov 06, 2022 11:54:44 AM io.opentelemetry.sdk.internal.ThrottlingLogger log
WARNING: Too many log messages detected. Will only log once per minute from now on.
Nov 06, 2022 11:54:44 AM io.opentelemetry.sdk.internal.ThrottlingLogger doLog
WARNING: Instrument jvm.memory.pool.committed has recorded multiple values for the same attributes.
Nov 06, 2022 11:54:44 AM io.opentelemetry.sdk.internal.ThrottlingLogger doLog
WARNING: Instrument jvm.memory.pool.init has recorded multiple values for the same attributes.
Nov 06, 2022 11:54:44 AM io.opentelemetry.sdk.internal.ThrottlingLogger doLog
WARNING: Instrument jvm.memory.pool.init has recorded multiple values for the same attributes.
Nov 06, 2022 11:54:44 AM io.opentelemetry.sdk.internal.ThrottlingLogger doLog
WARNING: Instrument jvm.memory.pool.init has recorded multiple values for the same attributes.
Nov 06, 2022 11:54:44 AM io.opentelemetry.sdk.internal.ThrottlingLogger doLog
WARNING: Instrument jvm.memory.pool.init has recorded multiple values for the same attributes.
Nov 06, 2022 11:54:44 AM io.opentelemetry.sdk.internal.ThrottlingLogger doLog
WARNING: Instrument jvm.memory.pool.init has recorded multiple values for the same attributes.
Nov 06, 2022 11:54:44 AM io.opentelemetry.sdk.internal.ThrottlingLogger log
WARNING: Too many log messages detected. Will only log once per minute from now on.
Nov 06, 2022 11:54:44 AM io.opentelemetry.sdk.internal.ThrottlingLogger doLog
WARNING: Instrument jvm.memory.pool.init has recorded multiple values for the same attributes.
Nov 06, 2022 11:54:44 AM io.opentelemetry.sdk.internal.ThrottlingLogger doLog
WARNING: Instrument jvm.memory.pool.max has recorded multiple values for the same attributes.
Nov 06, 2022 11:54:44 AM io.opentelemetry.sdk.internal.ThrottlingLogger doLog
WARNING: Instrument jvm.memory.pool.max has recorded multiple values for the same attributes.
Nov 06, 2022 11:54:44 AM io.opentelemetry.sdk.internal.ThrottlingLogger doLog
WARNING: Instrument jvm.memory.pool.max has recorded multiple values for the same attributes.
Nov 06, 2022 11:54:44 AM io.opentelemetry.sdk.internal.ThrottlingLogger doLog
WARNING: Instrument jvm.memory.pool.max has recorded multiple values for the same attributes.
Nov 06, 2022 11:54:44 AM io.opentelemetry.sdk.internal.ThrottlingLogger doLog
WARNING: Instrument jvm.memory.pool.max has recorded multiple values for the same attributes.
Nov 06, 2022 11:54:44 AM io.opentelemetry.sdk.internal.ThrottlingLogger log
WARNING: Too many log messages detected. Will only log once per minute from now on.
Nov 06, 2022 11:54:44 AM io.opentelemetry.sdk.internal.ThrottlingLogger doLog
WARNING: Instrument jvm.memory.pool.max has recorded multiple values for the same attributes.
Nov 06, 2022 11:54:44 AM io.opentelemetry.sdk.internal.ThrottlingLogger doLog
WARNING: Instrument jvm.memory.pool.used has recorded multiple values for the same attributes.
Nov 06, 2022 11:54:44 AM io.opentelemetry.sdk.internal.ThrottlingLogger doLog
WARNING: Instrument jvm.memory.pool.used has recorded multiple values for the same attributes.
Nov 06, 2022 11:54:44 AM io.opentelemetry.sdk.internal.ThrottlingLogger doLog
WARNING: Instrument jvm.memory.pool.used has recorded multiple values for the same attributes.
Nov 06, 2022 11:54:44 AM io.opentelemetry.sdk.internal.ThrottlingLogger doLog
WARNING: Instrument jvm.memory.pool.used has recorded multiple values for the same attributes.
Nov 06, 2022 11:54:44 AM io.opentelemetry.sdk.internal.ThrottlingLogger doLog
WARNING: Instrument jvm.memory.pool.used has recorded multiple values for the same attributes.
Nov 06, 2022 11:54:44 AM io.opentelemetry.sdk.internal.ThrottlingLogger log
WARNING: Too many log messages detected. Will only log once per minute from now on.
Nov 06, 2022 11:54:44 AM io.opentelemetry.sdk.internal.ThrottlingLogger doLog
WARNING: Instrument jvm.memory.pool.used has recorded multiple values for the same attributes.
Nov 06, 2022 11:54:44 AM io.opentelemetry.sdk.internal.ThrottlingLogger doLog
WARNING: Instrument jvm.threads.count has recorded multiple values for the same attributes.

But then in the Kibana UI

But it is shipping something

stephenb · November 7, 2022, 12:05am

@User28

@alexander.wert Feel Free to correct me...

Turns out my Config Does Work.

Note I did use 8.5 so I have 7.17.4

To Gather JVM Metrics with OTEL you need to use the contrib package and collect via JMX.
Secondly you will need to build a couple visualizations as OTEL / JVM / JMX does not deliver the metrics in a compatible form with our Curated UI for JVM Metrics.

I did in fact validate I am gathering metrics and the metrics in the APM data View ...
So in the mean time you would need to build some visualizations on your own.
Stay Tuned.

Yup I can pretty much build the Viz... Here is the Heap

User28 · November 7, 2022, 5:08pm

@stephenb Thank you for sharing info

I managed to get memory metrics by just adding OTEL_METRICS_EXPORTER:otlp in the environment of the application

Note: I 'm using OTel collector as side car to monitor the application.

But somehow i don't see CPU usage and System memory usage

Otel Collector yaml :

apiVersion: opentelemetry.io/v1alpha1
kind: OpenTelemetryCollector
metadata:
  name: designer-sidecar
spec:
  image: otel/opentelemetry-collector-contrib-dev:latest
  mode: sidecar
  config: |
    
    receivers:
      otlp:
        protocols:
          http:
          grpc:
      hostmetrics:
        collection_interval: 30s
        scrapers:
          cpu:
          memory:
          load:
            cpu_average: true
          disk:
          filesystem:
          network:
          paging:
          processes:
    
    exporters:
      logging:
      otlp/elastic:
        endpoint: http://efk-dev-apm-http.efk-dev.svc:8200
        headers: 
          Authorization: Bearer asdasdsadsad
        tls:
          insecure: true
    
    service:
      pipelines:
        traces:
          receivers: [otlp]
          exporters: [logging, otlp]
        metrics:
          receivers: [otlp]
          exporters: [logging, otlp]
        logs: 
          receivers: [otlp]
          exporters: [logging, otlp]

stephenb · November 7, 2022, 6:10pm

Hi @User28

2 Things

1st I am not an expert on sidecars etc but it looks to me you are not using the the JMX metrics gatherer, my understanding today that is the only way to get JVM metrics see here

2nd Even after you get that working Today, the JVM metrics that OTEL creates are not directly compatible with the fields in the Elastic APM App so you would need to build your own dashboards... Our APM Team is thinking about Compatibility for metrics but for the short term they are not. (I think you are getting the Heap / Non Heap Memory "By Coincidence")

Perhaps I could help with some viz dashboard at some point after you get the data

After you get the data in... you should be able to see it in Discover I suspect if you went to discover you would not see that data...

So get that data in... then we will take a look at Viz... If you have it already great!

User28 · November 9, 2022, 6:28pm

Hi @stephenb ,
I still didn't got JVM data.

Can you please share how you have used contrib to make it work. Did not find any example:

Thanks

stephenb · November 9, 2022, 9:02pm

Where are you looking for the Data? In Discover?

Also you are doing this in K8s I think I have no clue how to set JMX in K8s.

Here is my simple example btw I am bvader on github

git clone https://github.com/bvader/cardatabase.git

cd cardatabase

curl -O  https://repo1.maven.org/maven2/co/elastic/apm/elastic-apm-agent/${AGENT_VERSION}/elastic-apm-agent-1.34.0.jar

./mvnw clean package -Dmaven.test.skip=true

export OTEL_RESOURCE_ATTRIBUTES=service.name=cardatabase-local-otel,service.version=1.1,deployment.environment=production,agent.id=hyperion,host.name=hyperion
export OTEL_EXPORTER_OTLP_ENDPOINT=https://sdfgsdfgsdf.apm.us-west2.gcp.elastic-cloud.com
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer sdfgsdfgdsfgdsfgsdfg"
export OTEL_METRICS_EXPORTER="otlp"
export OTEL_LOGS_EXPORTER="otlp"
java -javaagent:./opentelemetry-javaagent.jar \
    -Dcom.sun.management.jmxremote \
    -Dcom.sun.management.jmxremote.port=9010 \
    -Dcom.sun.management.jmxremote.local.only=false \
    -Dcom.sun.management.jmxremote.authenticate=false \
    -Dcom.sun.management.jmxremote.ssl=false \
    -Djava.rmi.server.hostname=localhost \
    -jar target/cardatabase-0.0.1-SNAPSHOT.jar

in another terminal

$ wget https://github.com/open-telemetry/opentelemetry-java-contrib/releases/download/v1.19.1/opentelemetry-jmx-metrics.jar

Create a file session.properties

otel.jmx.service.url = service:jmx:rmi:///jndi/rmi://localhost:9010/jmxrmi
otel.jmx.target.system = jvm
otel.jmx.interval.milliseconds = 500
otel.metrics.exporter = otlp
otel.exporter.otlp.endpoint = https://sdfgsdfgsdf.apm.us-west2.gcp.elastic-cloud.com

$ java -jar opentelemetry-jmx-metrics.jar -config ./session.properties 
Nov 09, 2022 1:00:22 PM io.opentelemetry.contrib.jmxmetrics.JmxMetrics start
INFO: Started GroovyRunner.
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by io.opentelemetry.contrib.jmxmetrics.JmxClient (file:/Users/sbrown/workspace/repos/bvader/tmp/cardatabase/opentelemetry-jmx-metrics.jar) to constructor com.sun.security.sasl.Provider()
WARNING: Please consider reporting this to the maintainers of io.opentelemetry.contrib.jmxmetrics.JmxClient
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
Nov 09, 2022 1:00:23 PM io.opentelemetry.sdk.internal.ThrottlingLogger doLog
WARNING: Instrument jvm.classes.loaded has recorded multiple values for the same attributes.
Nov 09, 2022 1:00:23 PM io.opentelemetry.sdk.internal.ThrottlingLogger doLog
WARNING: Instrument jvm.gc.collections.count has recorded multiple values for the same attributes.
Nov 09, 2022 1:00:23 PM io.opentelemetry.sdk.internal.ThrottlingLogger doLog
WARNING: Instrument jvm.gc.collections.count has recorded multiple values for the same attribu

This seems pretty experimental at this point in my opinion.

if you run this several time it will create some transactions...

$ curl localhost:8080/api/cars

Then you should be able to see data as I showed Above this is with Elastic Stack 8.5

system · November 30, 2022, 5:03pm

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Missing JVM Metrics with Opentelemetry APM java	2	1391	October 17, 2022
Kibana APM metrics view din't show Opentelemetry JVM metrics APM java , open-telemetry	11	726	March 18, 2024
Elastic APM Agent and OpenTelemetry Agent Issue APM java	7	382	December 15, 2023
Opentelemetry migration APM java , server	3	712	July 13, 2020
OpenTelemetry metrics are not properly shown on Kibana APM docker , java , open-telemetry	3	803	March 7, 2023

No JVMs were found APM

Related topics