New GC metrics in Elastic 8.10

Elastic Observability APM
,
1

We are using elastic stack 8.10.1 and elastic java agent 1.42.
With new changes to GC collection and memory pools usage, would like to clarify the following

  1. GC collection count per min is too high (i see 510000 per min)
  2. GC collection time per min is also really high (8,600,500 per min)

Is there any document with description on what it means. For GC collection time, is it collection GC pause time (excluding concurrent mark time as application will be running during concurrent phase)

image
image3208×824 328 KB

Heap memory usage by pool shows only old generation usage.

image
image1602×818 136 KB

2

Is that showing the combined metrics of several services with the same service name?

3

First issue is its not filtering when i navigate from instances list in APM service overview page. I need to click on metrics tab to get the new view with additional metrics
Then i see node name as filter(it should be container or pod in description). Metrics not available with single select. Once i select more than 1, started seeing metrics with really high number. Once i change time range to longer duration, numbers are down to 3 digits.
Could someone clarify or link to document with metrics description for CPU, system Memory(is it cgroup stats against container memory limit), GC heap memory, GC time.
Particularly with GC time its not clear on what it means. is it application pause time, how its been aggregated (Average?) with long duration?

4

The GC times are the ones produced by GarbageCollectorMXBean.getCollectionTime(). The details are provided with that method. Broadly, this should be pause times for the young generation but full cycle times for the old gen (so the old gen times tend not to be that useful for analysis). These are correct in the elastic documents, but they are accumulated times not steps, and it's possible that they don't display correctly when there are multiple services being shown, hence my question, it will help for me to try and reproduce. From your answer it looks like you are indeed looking at stats for multiple services in one graph. I'll look at whether there are issues for that scenario

5

Thanks Jack. I am able to filter down to particular service. Looks like navigating from service overview page to JVM stats by click on instance link from overview page, navigate to old JVM metrics view. Need to click explicitly on metric tab to get new view and then need to search and select container/JVM to get new GC metrics. Is this expected behavior?

6

To me that indeed looks like a UI bug. I'll open a Ticket with the UI Team, thanks for reporting!

7

Here's the corresponding Issue you can follow.

8

thanks

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.