APM Java Agent JMX Metrics for Springboot application (capture_jmx_metrics)

[chandra2037 (Chandra)]

Kibana version: 7.4.0

Elasticsearch version: 7.4.0

APM Server version: 7.4.0

APM Agent language and version: Java and 1.10.0

Trying to get metrics from JMX for Spring Boot microservice which is deployed in Kubernetes. Enabled Spring actuator where I can see the metrics using "/actuator" endpoint.

Included the following configuration in elasticapm.properties file

capture_jmx_metrics="object_name[com.zaxxer.hikari:type=Pool,name=*] attribute[TotalConnections] attribute[ActiveConnections]"

I am expecting Hikari connection pool metrics in Elasticsearch, but not seeing anything. By default APM Javagent reporting other metrics like JVM, GC, transaction performance metrics.

Am I missing something, or do I need to configure anything to capture Hikari Pool connections metrics.

Thank you for any help.

[felixbarny (Felix Barnsteiner)]

Hi and thanks for your question!

Could you send a screenshot of a JMX console or similar which shows the metric? Admittedly, we could do a better job in terms of logging if a JMX value could not be found. Have you tried debugging JmxMetricTracker?

Another reason which comes to mind is that maybe we are trying to discover the JMX metrics before they are registered.

To check if that's the case start your application without the capture_jmx_metrics option. Then add the value at runtime and wait for half a minute. Note that this does not work if the elasticapm.properties are in src/main/resources but only if they are in the same folder as the agent jar. Another option is to call System.setProperty("co.elastic.apm.capture_jmx_metrics", "object_name[com.zaxxer.hikari:type=Pool,name=*] attribute[TotalConnections] attribute[ActiveConnections]") at runtime (for example by evaluating an expression while debugging).

Also, don't quote the value in the property file.

Hope that helps
/Felix

[chandra2037 (Chandra)]

Thank you for your response.

Running the application in a Kubernetes Pod, I am not able to replace the elasticapm.properties file at runtime, getting "resource busy" error.

But Tried by disabling capture_jmx_metrics option, then calling System.setProperty("co.elastic.apm.capture_jmx_metrics", "object_name[com.zaxxer.hikari:type=Pool,name=*] attribute[TotalConnections] attribute[ActiveConnections]") at runtime. No luck, not getting any additional metrics.

Providing the screenshots of Spring boot actuator metrics call
Calling ...../api/v1/chandratestservice/actuator/metrics/ endpoint gets me

Now in order to get individual metrics I will request ...../api/v1/chandratestservice/actuator/metrics/hikaricp.connections.active endpoint, which gets me

Not sure how to debug JmxMetricTracker for an application deployed in Kubernetes Pod.

I have a feeling that elasticapm configuration needs to be tweaked or I will get not these kind of metrics from APM agent.

Thank you for any input on this.

[felixbarny (Felix Barnsteiner)]

I've created a PR which both improves logging (set log_level=debug to see it) and can also cope with MBeans which were added after the agent starts: https://github.com/elastic/apm-agent-java/pull/879

Could you test if these changes work for you? The build artifacts are in https://apm-ci.elastic.co/job/apm-agent-java/job/apm-agent-java-mbp/job/PR-879/3/artifact/src/github.com/elastic/apm-agent-java/ (note this link expires after a while).

[chandra2037 (Chandra)]

Tested with the changes with log_level=debug but no luck. And I am not seeing anything in the logs also. What kind of log statements can I expect?

I tried Micrometer Elasticsearch implementation to report metrics directly from Micrometer to Elasticsearch. It's working, getting all the metrics. But it won't report the contextual information like from which service, host, etc.. It would have been great if APM agent able to report the metrics.

Not sure what I am missing.

To confirm if APM agent reports the JMX metrics, it will be indexed in Elasticsearch with processor.event : "metric" right, and will all new metrics be stored as new fields?

[felixbarny (Felix Barnsteiner)]

Could you share the debug logs? I expect that the object name does not match the actual mbean.

We do have plans for better integration with Micrometer.

[chandra2037 (Chandra)]

Here are the debug logs

Thank you.

[felixbarny (Felix Barnsteiner)]

I found another bug which prevented the registration of JMX metrics

Could you try again? https://apm-ci.elastic.co/job/apm-agent-java/job/apm-agent-java-mbp/job/PR-879/5/artifact/src/github.com/elastic/apm-agent-java/

I have successfully tested it manually now with

capture_jmx_metrics=object_name[com.zaxxer.hikari:type=HikariDataSource,name=*] attribute[ConnectionTimeout]

You should see logs similar to this:

DEBUG co.elastic.apm.agent.jmx.JmxMetricTracker - MBean added at runtime: com.zaxxer.hikari:type=HikariDataSource,name=*
DEBUG co.elastic.apm.agent.jmx.JmxMetricTracker - Found mbeans for object name com.zaxxer.hikari:type=HikariDataSource,name=*
DEBUG co.elastic.apm.agent.jmx.JmxMetricTracker - Found number attribute ConnectionTimeout=30000
DEBUG co.elastic.apm.agent.jmx.JmxMetricTracker$JmxMetricRegistration - Registering JMX metric com.zaxxer.hikari:name=dataSource,type=HikariDataSource ConnectionTimeout.null as metric_name: jvm.jmx.ConnectionTimeout labels: name=dataSource, type=HikariDataSource

You'll find the metrics in the index apm-*-metric

field name	value
labels.name	dataSource
labels.type	HikariDataSource
jvm.jmx.ConnectionTimeout	30000

[chandra2037 (Chandra)]

Thank you @felixbarny. I tried with the config you tested with, it's working. But when I tried the following config to get HikariCP pool metrics, not getting any metrics.

object_name[com.zaxxer.hikari:type=Pool,name=HikariPool-1] attribute[TotalConnections]

Here are the debug logs.

Tested to make sure that we can access HikariCP Pool metrics, tried below code as described here

MBeanServer mBeanServer = ManagementFactory.getPlatformMBeanServer();
ObjectName poolName = new ObjectName("com.zaxxer.hikari:type=Pool (HikariPool-1)");
HikariPoolMXBean poolProxy = JMX.newMXBeanProxy(mBeanServer, poolName, HikariPoolMXBean.class);
int idleConnections = poolProxy.getIdleConnections();
int totalConnections = poolProxy.getTotalConnections();

I am looking to get Connection Pool connections metrics (activeConnections, totalConnections, idleConnections)

Thanks again for your help.

[felixbarny (Felix Barnsteiner)]

Have you tried object_name[com.zaxxer.hikari:type=Pool (HikariPool-1)] attribute[TotalConnections]?

[chandra2037 (Chandra)]

@felixbarny You are awesome! I totally missed this config

It's working now. Now, I am getting the Pool metrics.

Question: When can I expect these changes in the release version?

Thank you very much for your time and help.

[felixbarny (Felix Barnsteiner)]

Sorry for the late reply, I was on PTO last week. We plan to release soon, best to subscribe the GitHub repository for releases in order to get notified.