# (APM java-agent) JMX metrics does not show up in apm-\* index

**URL:** https://discuss.elastic.co/t/apm-java-agent-jmx-metrics-does-not-show-up-in-apm-index/224048
**Category:** APM
**Tags:** java
**Created:** [March 18, 2020, 8:40am UTC](https://discuss.elastic.co/t/apm-java-agent-jmx-metrics-does-not-show-up-in-apm-index/224048 "2020-03-18T08:40:49Z")
**Posts on this page:** 7
**Page:** 1

<div class="post-metadata">

### Author: ![joek](https://avatars.discourse-cdn.com/v4/letter/j/96bed5/32.png) [@joek](https://discuss.elastic.co/u/joek)
#### Post date: [March 18, 2020, 8:40am UTC](https://discuss.elastic.co/t/apm-java-agent-jmx-metrics-does-not-show-up-in-apm-index/224048/1 "2020-03-18T08:40:49Z")

</div>

**Kibana version** :  
kibana-7.6.1-1.x86\_64  
**Elasticsearch version** :  
elasticsearch-7.6.1-1.x86\_64  
**APM Server version** :  
apm-server-7.6.1-1.x86\_64  
**APM Agent language and version** :  
elastic-apm-agent-1.14.0.jar  
**Browser version** :  
Chrome any  
**Original install method (e.g. download page, yum, deb, from source, etc.) and version**:  
yum except for agent  
**Fresh install or upgraded from other version?**  
fresh insall  
**Is there anything special in your setup?** For example, are you using the Logstash or Kafka outputs? Are you using a load balancer in front of the APM Servers? Have you changed index pattern, generated custom templates, changed agent configuration etc.  
n/a  
**Description of the problem including expected versus actual behavior. Please include screenshots (if relevant)**:

**Steps to reproduce** :

1. Configure tomcat to use APM-agent - works fine - APM metrics are visible in Kibana
2. Configure some capture\_jmx\_metrics - does not show up in index
3. 

Config: (service name is correctly parsed from elasticapm.properties )  
`capture_jmx_metrics=object_name[com.zaxxer.hikari:type=Pool (HikariPool-*)] attribute[TotalConnections]`  
`service_name=Tomcat7_APM_Agent`  
This works:  
` ./check_jmx -U service:jmx:rmi:///jndi/rmi://logserver.je.local:2222/jmxrmi -O com.zaxxer.hikari:type="Pool (HikariPool-*)" -A TotalConnections`  
JMX OK - TotalConnections = 9 | 'TotalConnections'=9;;;😉

(Disabling std JMX did not make any difference)  
**Errors in browser console (if relevant)**:  
n/a  
**Provide logs and/or server output (if relevant)**:  
no jmx data shows up. JMX data verified with JConsole and misc cmdline tools  
Only PS Scavenge and PS Marksweep visible in labels.name

---

<div class="post-metadata">

### Author: ![Eyal\_Koren](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/eyal_koren/32/36830_2.png) [@Eyal\_Koren](https://discuss.elastic.co/u/Eyal_Koren)
#### Post date: [March 18, 2020, 9:17am UTC](https://discuss.elastic.co/t/apm-java-agent-jmx-metrics-does-not-show-up-in-apm-index/224048/2 "2020-03-18T09:17:33Z")

</div>

Hi and welcome to the forum.

If I understand correctly, the ObjectName's type property value is of this form: `"Pool (HikariPool-*)"` (meaning, the `(HikariPool-*)` is part of the `type` property and not `name` property).  
Did you try configuring: `capture_jmx_metrics=object_name[com.zaxxer.hikari:type="Pool (HikariPool-*)"] attribute[TotalConnections]`?

If that doesn't work, please set [`log_level`](https://www.elastic.co/guide/en/apm/agent/java/current/config-logging.html#config-log-level) to `DEBUG` and upload the resulted log, maybe we can find hints there.

I hope this helps.

---

<div class="post-metadata">

### Author: ![joek](https://avatars.discourse-cdn.com/v4/letter/j/96bed5/32.png) [@joek](https://discuss.elastic.co/u/joek)
#### Post date: [March 18, 2020, 10:12am UTC](https://discuss.elastic.co/t/apm-java-agent-jmx-metrics-does-not-show-up-in-apm-index/224048/3 "2020-03-18T10:12:59Z")

</div>

Hi, Thanks for quick reply 🙂  
I changed config as instructed and restarted in DEBUG mode.  
Still no Hikari-stuff visible in Kibana.  
Could not attach logfile so I attach some lines:

# egrep -i "jmx" /usr/share/tomcat/apmagent.log

`2020-03-18 10:53:27.111 [main] INFO co.elastic.apm.agent.util.JmxUtils - Found JVM-specific OperatingSystemMXBean interface: com.sun.management.OperatingSystemMXBean`  
`2020-03-18 10:53:27.424 [main] DEBUG co.elastic.apm.agent.configuration.StartupInfo - capture_jmx_metrics: 'object_name[com.zaxxer.hikari:type="Pool (HikariPool-*)"] attribute[TotalConnections]' (source: /opt/apm-agent/lib/elasticapm.properties)`  
`2020-03-18 10:53:27.582 [main] DEBUG co.elastic.apm.agent.jmx.JmxMetricTracker - Init JMX metric tracking with server com.sun.jmx.mbeanserver.JmxMBeanServer@1376c05c`  
`2020-03-18 10:53:27.585 [main] DEBUG co.elastic.apm.agent.jmx.JmxMetricTracker - Found mbeans for object name com.zaxxer.hikari:type="Pool (HikariPool-*)"`

Other lines:  
`2020-03-18 10:53:46.739 [http-bio-8080-exec-2] DEBUG co.elastic.apm.agent.bci.ElasticApmAgent - Method match for instrumentation ExecuteBatchInstrumentation: (((name(equals(executeBatch)) or name(equals(executeLargeBatch))) and hasParameter(ofSize(0))) and isPublic()) matches public long[] com.zaxxer.hikari.pool.ProxyStatement.executeLargeBatch() throws java.sql.SQLException`

`2020-03-18 10:53:52.213 [HikariPool-1 connection adder] DEBUG co.elastic.apm.agent.impl.transaction.AbstractSpan - increment references to '' 00-d2bbd04e712c3144c09b9fefdff3b445-4cec43b361033532-01 (220d35d) (4)`  
`2020-03-18 10:53:52.214 [HikariPool-1 connection adder] DEBUG co.elastic.apm.agent.impl.ElasticApmTracer - Activating '' 00-d2bbd04e712c3144c09b9fefdff3b445-4cec43b361033532-01 (220d35d) on thread 68`  
`2020-03-18 10:53:52.238 [HikariPool-1 connection adder] DEBUG co.elastic.apm.agent.impl.transaction.AbstractSpan - increment references to '' 00-d2bbd04e712c3144c09b9fefdff3b445-4cec43b361033532-01 (220d35d) (8)`

---

<div class="post-metadata">

### Author: ![joek](https://avatars.discourse-cdn.com/v4/letter/j/96bed5/32.png) [@joek](https://discuss.elastic.co/u/joek)
#### Post date: [March 18, 2020, 11:00am UTC](https://discuss.elastic.co/t/apm-java-agent-jmx-metrics-does-not-show-up-in-apm-index/224048/4 "2020-03-18T11:00:10Z")

</div>

Hi,  
Reverted to my original config and extended it a bit and from agent it looks OK (?)

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/2/0/202fd5d49fdc915006c89d43e8aff128542172ef.png)

At least it says:  
`MBean added at runtime: com.zaxxer.hikari:type=Pool (HikariPool-*)`  
`Found mbeans for object name com.zaxxer.hikari:type=Pool (HikariPool-*)`  
`Found number attribute TotalConnections=1`  
`Found number attribute ActiveConnections=0`  
`Found number attribute IdleConnections=1`  
`Found number attribute ThreadsAwaitingConnection=0`

`Registering JMX metric com.zaxxer.hikari:type=Pool (HikariPool-1) TotalConnections.null as metric_name: jvm.jmx.TotalConnections labels: type=Pool (HikariPool-1)`  
`Registering JMX metric com.zaxxer.hikari:type=Pool (HikariPool-1) ActiveConnections.null as metric_name: jvm.jmx.ActiveConnections labels: type=Pool (HikariPool-1)`

and so on ....

But still nothing visible in Kibana (std APM metrics gets delivered)

---

<div class="post-metadata">

### Author: ![Eyal\_Koren](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/eyal_koren/32/36830_2.png) [@Eyal\_Koren](https://discuss.elastic.co/u/Eyal_Koren)
#### Post date: [March 18, 2020, 11:16am UTC](https://discuss.elastic.co/t/apm-java-agent-jmx-metrics-does-not-show-up-in-apm-index/224048/5 "2020-03-18T11:16:22Z")

</div>

Yes, this looks OK.  
Custom metrics would not show up automatically in Kibana, but they are stored in Elasticsearch as explained in documentation and you can create a custom visualization for them.

---

<div class="post-metadata">

### Author: ![joek](https://avatars.discourse-cdn.com/v4/letter/j/96bed5/32.png) [@joek](https://discuss.elastic.co/u/joek)
#### Post date: [March 18, 2020, 11:20am UTC](https://discuss.elastic.co/t/apm-java-agent-jmx-metrics-does-not-show-up-in-apm-index/224048/6 "2020-03-18T11:20:18Z")

</div>

Hi,  
OK, then I'll follow the Kibana trail.  
My final config for reference:  
`capture_jmx_metrics=object_name[com.zaxxer.hikari:type=Pool (HikariPool-*)] attribute[TotalConnections] attribute[ActiveConnections] attribute[IdleConnections] attribute[ThreadsAwaitingConnection]`

Thanks for guiding !

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [April 8, 2020, 7:20am UTC](https://discuss.elastic.co/t/apm-java-agent-jmx-metrics-does-not-show-up-in-apm-index/224048/7 "2020-04-08T07:20:25Z")

</div>

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.
