Java APM1.41.0 JVM used is Zero

Elastic Observability APM
1

image
image1410×574 41.8 KB

image
image1418×590 40.5 KB

1.38.0 can catch the jvm used
Programs typically use around 200-300M of heap memory
Non-Heap memory is 260~320M in 1.38.0 befor

JDK:
JDK17.0.8

Kibana version:
7.13.2
Elasticsearch version:
7.13.2
APM Server version:
7.13.2

APM Agent language and version:
1.41.0

2

Hi, i try the new 1.41.1 but the displayed data is still incorrect
image

image
image1496×116 6.15 KB

It seems that the data is exactly 10 times different

3

the bug is the same with Bug: APM Java - Springboot, v1.39.0+ - No JVM Metrics Found - #12 by Jonas_Kunz

4

Hi,

I've tried to reproduce with the same stack version and Java agent version and it seems that metrics are properly captured, however, the rounding in the UI could definitely cause some confusion here as overlay is shown with the biggest unit (here GB).

  • you likely have set a limit to 1.3gb with -Xmx, thus the overlay uses gigabyte as units.
  • the application currently have a very small usage of heap (less than 100mb), thus it gets rounded to 0.0 GB in the overlay.

Could you try to let the application run for a while with some activity so we can see how does the metrics look like when the heap usage goes over 100mb ?
Alternatively, could you try to remove the -Xmx argument from your JVM ? with that the heap usage should be shown in megabytes instead of gigabytes, hence there shouldn't be any rounding here.

5

I try another program.

This does not use the disable_metrics parameter, XMX3250
And the Programe is running more then one hour(used about 700M memory+-)

image
image2852×620 45.9 KB

this is use the -Delastic.apm.disable_metrics=jvm.fd.*,jvm.memory.non_heap.pool.*
XMX=3086, used 500M~2G

image
image2010×576 63.3 KB

JVM

-Xms1024m -Xmx3086m
-XX:+UseContainerSupport 
-XX:MinRAMPercentage=50.0 -XX:MaxRAMPercentage=80.0 
-Delastic.apm.log_level=ERROR 
-Delastic.apm.disable_metrics=jvm.fd.*,jvm.memory.non_heap.pool.* 
-Delastic.apm.service_node_name=xxx
-Delastic.apm.service_name=xxxxx
-Delastic.apm.transaction_sample_rate=0.8

image
image748×705 28 KB

I deploy it in a container, and in order to prevent memory overflow, XMX is calculated. In the production environment, I am currently afraid to delete XMX

There also can not display jvm used

this is apm 1.38.0

image
image2300×530 66.5 KB

the data is same with the jmap

I will run it for a day first to see if there will be any changes when the usage exceeds 1G

6

I search the apm-metric index
when apm is 1.38.0
the promram real used about 1.9G

"non_heap" : {
      "committed" : 3.74669312E8,
      "max" : 9.77272832E8,
      "used" : 3.663926E8
    },
    "heap" : {
      "committed" : 2.76824064E9,
      "max" : 3.235905536E9,
      "used" : 1.956814336E9
    }

when apm is 1.41.x
the program real used 1.2G used = 1078312208 (1028.358657836914MB)
but data is 3.45344E7 used

    "non_heap" : {
      "committed" : 3.4930688E7,
      "max" : 9.89855744E8,
      "used" : 3.1833088E7
    },
    "heap" : {
      "committed" : 1.077936128E9,
      "max" : 3.235905536E9,
      "used" : 3.45344E7
    }
7

Hi,

I've managed to reproduce it, and that's due to a regression introduced in 1.39.0 which hasn't been reported yet, so congrats for catching this !

I've opened Fix JVM memory metrics regression by SylvainJuge · Pull Request #3279 · elastic/apm-agent-java · GitHub to fix it and it's already merged thus will be included in the next release.

So in your case you should either:

8

Thank you very much. Using the snapshot version, the issue has been corrected

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.