# APM Java Agent JMX Metrics for Springboot application (capture\_jmx\_metrics)
**URL:** https://discuss.elastic.co/t/apm-java-agent-jmx-metrics-for-springboot-application-capture-jmx-metrics/202700
**Category:** APM
**Tags:** java
**Created:** [October 8, 2019, 4:21pm UTC](https://discuss.elastic.co/t/apm-java-agent-jmx-metrics-for-springboot-application-capture-jmx-metrics/202700 "2019-10-08T16:21:54Z")
**Posts on this page:** 13
**Page:** 1
### Author:  [@chandra2037](https://discuss.elastic.co/u/chandra2037)
#### Post date: [October 8, 2019, 4:21pm UTC](https://discuss.elastic.co/t/apm-java-agent-jmx-metrics-for-springboot-application-capture-jmx-metrics/202700/1 "2019-10-08T16:21:55Z")
**Kibana version** : 7.4.0
**Elasticsearch version** : 7.4.0
**APM Server version** : 7.4.0
**APM Agent language and version** : Java and 1.10.0
Trying to get metrics from JMX for Spring Boot microservice which is deployed in Kubernetes. Enabled Spring actuator where I can see the metrics using "/actuator" endpoint.
Included the following configuration in elasticapm.properties file
`capture_jmx_metrics="object_name[com.zaxxer.hikari:type=Pool,name=*] attribute[TotalConnections] attribute[ActiveConnections]"`
I am expecting Hikari connection pool metrics in Elasticsearch, but not seeing anything. By default APM Javagent reporting other metrics like JVM, GC, transaction performance metrics.
Am I missing something, or do I need to configure anything to capture Hikari Pool connections metrics.
Thank you for any help.
---
### Author:  [@felixbarny](https://discuss.elastic.co/u/felixbarny)
#### Post date: [October 8, 2019, 5:19pm UTC](https://discuss.elastic.co/t/apm-java-agent-jmx-metrics-for-springboot-application-capture-jmx-metrics/202700/2 "2019-10-08T17:19:59Z")
Hi and thanks for your question!
Could you send a screenshot of a JMX console or similar which shows the metric? Admittedly, we could do a better job in terms of logging if a JMX value could not be found. Have you tried debugging [`JmxMetricTracker`](https://github.com/elastic/apm-agent-java/blob/v1.10.0/apm-agent-plugins/apm-jmx-plugin/src/main/java/co/elastic/apm/agent/jmx/JmxMetricTracker.java)?
Another reason which comes to mind is that maybe we are trying to discover the JMX metrics before they are registered.
To check if that's the case start your application without the `capture_jmx_metrics` option. Then add the value at runtime and wait for half a minute. Note that this does not work if the `elasticapm.properties` are in `src/main/resources` but only if they are in the same folder as the agent jar. Another option is to call `System.setProperty("co.elastic.apm.capture_jmx_metrics", "object_name[com.zaxxer.hikari:type=Pool,name=*] attribute[TotalConnections] attribute[ActiveConnections]")` at runtime (for example by evaluating an expression while debugging).
Also, don't quote the value in the property file.
Hope that helps
/Felix
---
### Author:  [@chandra2037](https://discuss.elastic.co/u/chandra2037)
#### Post date: [October 8, 2019, 7:41pm UTC](https://discuss.elastic.co/t/apm-java-agent-jmx-metrics-for-springboot-application-capture-jmx-metrics/202700/3 "2019-10-08T19:41:47Z")
Thank you for your response.
Running the application in a Kubernetes Pod, I am not able to replace the `elasticapm.properties` file at runtime, getting "resource busy" error.
But Tried by disabling `capture_jmx_metrics` option, then calling `System.setProperty("co.elastic.apm.capture_jmx_metrics", "object_name[com.zaxxer.hikari:type=Pool,name=*] attribute[TotalConnections] attribute[ActiveConnections]")` at runtime. No luck, not getting any additional metrics.
Providing the screenshots of Spring boot actuator metrics call
Calling `...../api/v1/chandratestservice/actuator/metrics/` endpoint gets me
Now in order to get individual metrics I will request `...../api/v1/chandratestservice/actuator/metrics/hikaricp.connections.active` endpoint, which gets me
Not sure how to debug JmxMetricTracker for an application deployed in Kubernetes Pod.
I have a feeling that elasticapm configuration needs to be tweaked or I will get not these kind of metrics from APM agent.
Thank you for any input on this.
---
### Author:  [@felixbarny](https://discuss.elastic.co/u/felixbarny)
#### Post date: [October 9, 2019, 3:15pm UTC](https://discuss.elastic.co/t/apm-java-agent-jmx-metrics-for-springboot-application-capture-jmx-metrics/202700/4 "2019-10-09T15:15:05Z")
I've created a PR which both improves logging (set `log_level=debug` to see it) and can also cope with MBeans which were added after the agent starts: [https://github.com/elastic/apm-agent-java/pull/879](https://github.com/elastic/apm-agent-java/pull/879)
Could you test if these changes work for you? The build artifacts are in [https://apm-ci.elastic.co/job/apm-agent-java/job/apm-agent-java-mbp/job/PR-879/3/artifact/src/github.com/elastic/apm-agent-java/](https://apm-ci.elastic.co/job/apm-agent-java/job/apm-agent-java-mbp/job/PR-879/3/artifact/src/github.com/elastic/apm-agent-java/) (note this link expires after a while).
---
### Author:  [@chandra2037](https://discuss.elastic.co/u/chandra2037)
#### Post date: [October 10, 2019, 2:38pm UTC](https://discuss.elastic.co/t/apm-java-agent-jmx-metrics-for-springboot-application-capture-jmx-metrics/202700/5 "2019-10-10T14:38:41Z")
Tested with the changes with `log_level=debug` but no luck. And I am not seeing anything in the logs also. What kind of log statements can I expect?
I tried [Micrometer Elasticsearch](https://micrometer.io/docs/registry/elastic) implementation to report metrics directly from Micrometer to Elasticsearch. It's working, getting all the metrics. But it won't report the contextual information like from which service, host, etc.. It would have been great if APM agent able to report the metrics.
Not sure what I am missing.
To confirm if APM agent reports the JMX metrics, it will be indexed in Elasticsearch with `processor.event : "metric"` right, and will all new metrics be stored as new fields?
---
### Author:  [@felixbarny](https://discuss.elastic.co/u/felixbarny)
#### Post date: [October 10, 2019, 4:39pm UTC](https://discuss.elastic.co/t/apm-java-agent-jmx-metrics-for-springboot-application-capture-jmx-metrics/202700/6 "2019-10-10T16:39:01Z")
Could you share the debug logs? I expect that the object name does not match the actual mbean.
We do have plans for better integration with Micrometer.
---
### Author:  [@chandra2037](https://discuss.elastic.co/u/chandra2037)
#### Post date: [October 10, 2019, 5:40pm UTC](https://discuss.elastic.co/t/apm-java-agent-jmx-metrics-for-springboot-application-capture-jmx-metrics/202700/7 "2019-10-10T17:40:32Z")
Here are the [debug logs](https://pastebin.com/x741rZ9r)
Thank you.
---
### Author:  [@felixbarny](https://discuss.elastic.co/u/felixbarny)
#### Post date: [October 11, 2019, 1:06pm UTC](https://discuss.elastic.co/t/apm-java-agent-jmx-metrics-for-springboot-application-capture-jmx-metrics/202700/8 "2019-10-11T13:06:35Z")
I found another bug which prevented the registration of JMX metrics 🤦♂️
Could you try again? [https://apm-ci.elastic.co/job/apm-agent-java/job/apm-agent-java-mbp/job/PR-879/5/artifact/src/github.com/elastic/apm-agent-java/](https://apm-ci.elastic.co/job/apm-agent-java/job/apm-agent-java-mbp/job/PR-879/5/artifact/src/github.com/elastic/apm-agent-java/)
I have successfully tested it manually now with
```auto
capture_jmx_metrics=object_name[com.zaxxer.hikari:type=HikariDataSource,name=*] attribute[ConnectionTimeout]
```
You should see logs similar to this:
```auto
DEBUG co.elastic.apm.agent.jmx.JmxMetricTracker - MBean added at runtime: com.zaxxer.hikari:type=HikariDataSource,name=*
DEBUG co.elastic.apm.agent.jmx.JmxMetricTracker - Found mbeans for object name com.zaxxer.hikari:type=HikariDataSource,name=*
DEBUG co.elastic.apm.agent.jmx.JmxMetricTracker - Found number attribute ConnectionTimeout=30000
DEBUG co.elastic.apm.agent.jmx.JmxMetricTracker$JmxMetricRegistration - Registering JMX metric com.zaxxer.hikari:name=dataSource,type=HikariDataSource ConnectionTimeout.null as metric_name: jvm.jmx.ConnectionTimeout labels: name=dataSource, type=HikariDataSource
```
You'll find the metrics in the index `apm-*-metric`
| field name | value |
| --- | --- |
| labels.name | dataSource |
| labels.type | HikariDataSource |
| jvm.jmx.ConnectionTimeout | 30000 |
---
### Author:  [@chandra2037](https://discuss.elastic.co/u/chandra2037)
#### Post date: [October 12, 2019, 3:04pm UTC](https://discuss.elastic.co/t/apm-java-agent-jmx-metrics-for-springboot-application-capture-jmx-metrics/202700/9 "2019-10-12T15:04:30Z")
Thank you @felixbarny. I tried with the config you tested with, it's working. But when I tried the following config to get [HikariCP pool metrics](https://github.com/brettwooldridge/HikariCP/wiki/MBean-(JMX)-Monitoring-and-Management), not getting any metrics.
`object_name[com.zaxxer.hikari:type=Pool,name=HikariPool-1] attribute[TotalConnections]`
Here are the [debug logs](https://pastebin.com/Y3xyzx3c).
Tested to make sure that we can access HikariCP Pool metrics, tried below code as described [here](https://github.com/brettwooldridge/HikariCP/wiki/MBean-(JMX)-Monitoring-and-Management)
```
MBeanServer mBeanServer = ManagementFactory.getPlatformMBeanServer();
ObjectName poolName = new ObjectName("com.zaxxer.hikari:type=Pool (HikariPool-1)");
HikariPoolMXBean poolProxy = JMX.newMXBeanProxy(mBeanServer, poolName, HikariPoolMXBean.class);
int idleConnections = poolProxy.getIdleConnections();
int totalConnections = poolProxy.getTotalConnections();
```
I am looking to get Connection Pool connections metrics (activeConnections, totalConnections, idleConnections)
Thanks again for your help.
---
### Author:  [@felixbarny](https://discuss.elastic.co/u/felixbarny)
#### Post date: [October 13, 2019, 4:16pm UTC](https://discuss.elastic.co/t/apm-java-agent-jmx-metrics-for-springboot-application-capture-jmx-metrics/202700/10 "2019-10-13T16:16:16Z")
Have you tried `object_name[com.zaxxer.hikari:type=Pool (HikariPool-1)] attribute[TotalConnections]`?
---
### Author:  [@chandra2037](https://discuss.elastic.co/u/chandra2037)
#### Post date: [October 14, 2019, 12:18am UTC](https://discuss.elastic.co/t/apm-java-agent-jmx-metrics-for-springboot-application-capture-jmx-metrics/202700/11 "2019-10-14T00:18:41Z")
@felixbarny You are awesome! I totally missed this config 🤦♂️
It's working now. Now, I am getting the Pool metrics.
Question: When can I expect these changes in the release version?
Thank you very much for your time and help.
---
### Author:  [@felixbarny](https://discuss.elastic.co/u/felixbarny)
#### Post date: [October 21, 2019, 9:21am UTC](https://discuss.elastic.co/t/apm-java-agent-jmx-metrics-for-springboot-application-capture-jmx-metrics/202700/12 "2019-10-21T09:21:21Z")
Sorry for the late reply, I was on PTO last week. We plan to release soon, best to subscribe the GitHub repository for releases in order to get notified.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [November 11, 2019, 5:21am UTC](https://discuss.elastic.co/t/apm-java-agent-jmx-metrics-for-springboot-application-capture-jmx-metrics/202700/13 "2019-11-11T05:21:29Z")
This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.