# No JMX metrics in JBoss EAP 7.3.0

**URL:** <https://discuss.elastic.co/t/no-jmx-metrics-in-jboss-eap-7-3-0/238438>\
**Category:** APM\
**Tags:** java\
**Created:** [June 24, 2020, 10:52am UTC](https://discuss.elastic.co/t/no-jmx-metrics-in-jboss-eap-7-3-0/238438 "2020-06-24T10:52:16Z")\
**Posts on this page:** 16\
**Page:** 1

<div class="post-metadata">

**Author:** ![sopgreg](https://avatars.discourse-cdn.com/v4/letter/s/df788c/32.png) [@sopgreg](https://discuss.elastic.co/u/sopgreg)\
**Post date:** [June 24, 2020, 10:52am UTC](https://discuss.elastic.co/t/no-jmx-metrics-in-jboss-eap-7-3-0/238438/1 "2020-06-24T10:52:16Z")

</div>

**Kibana version** : 7.7.1

**Elasticsearch version** : 7.7.1

**APM Server version** : 7.7.1

**APM Agent language and version** : Java, 1.17

**Browser version** : not relevant

**Original install method (e.g. download page, yum, deb, from source, etc.) and version**: download

**Fresh install or upgraded from other version?** fresh install

**Is there anything special in your setup?** no, plain "quick setup" installation

**Description of the problem including expected versus actual behavior. Please include screenshots (if relevant)**:

The Java agent does not capture JMX metrics as defined like this in JBoss EAP 7.3.0:

`capture_jmx_metrics=object_name[jboss.as:subsystem=datasources,data-source=NomTDS_Pool,statistics=pool] attribute[AvailableCount] attribute[ActiveCount] attribute[InUseCount] attribute[TimedOut] attribute[WaitCount]`

Also tried with wildcard for the pool name:

`capture_jmx_metrics=object_name[jboss.as:subsystem=datasources,data-source=*,statistics=pool] attribute[AvailableCount] attribute[ActiveCount] attribute[InUseCount] attribute[TimedOut] attribute[WaitCount]`

The only logs that appear can be seen below. JBoss/Wildfly creates the ObjectName at runtime when it discovers the various data sources in the standalone.xml configuration. See screenshot from JConsole, the MBean is available with all the attributes defined:

![grafik](https://us1.discourse-cdn.com/elastic/original/3X/8/4/846537c3e816fdbef7d83ee754d1db43c9cb4b10.png)

![grafik](https://us1.discourse-cdn.com/elastic/original/3X/1/1/114da50c6e4604d6733ac8799abbcae1b6615258.png)

**Steps to reproduce** :

1. Start JBoss with Agent via --javagent (standalone.conf.bat)
2. Use configuration above

**Provide logs and/or server output (if relevant)**:

These are the only logs related to "jmx" in JBoss:

```
2020-06-23 17:14:35.172 [main] DEBUG co.elastic.apm.agent.configuration.StartupInfo - capture_jmx_metrics: 'object_name[jboss.as:subsystem=datasources,data-source=NomTDS_Pool,statistics=pool] attribute[AvailableCount] attribute[ActiveCount] attribute[InUseCount] attribute[TimedOut] attribute[WaitCount]' (source: C:\Server\JBoss_NomT\bin/elasticapm.properties)
2020-06-23 17:14:36.414 [main] DEBUG co.elastic.apm.agent.jmx.JmxMetricTracker - Deferring initialization of JMX metric tracking until log manager is initialized

2020-06-23 17:19:12.350 [elastic-apm-jmx-init] DEBUG co.elastic.apm.agent.jmx.JmxMetricTracker - Init JMX metric tracking with server com.sun.jmx.mbeanserver.JmxMBeanServer@402f88d
2020-06-23 17:19:12.356 [elastic-apm-jmx-init] DEBUG co.elastic.apm.agent.jmx.JmxMetricTracker - Found mbeans for object name jboss.as:subsystem=datasources,data-source=NomTDS_Pool,statistics=pool
```

---

<div class="post-metadata">

**Author:** ![felixbarny](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/felixbarny/32/27341_2.png) [@felixbarny](https://discuss.elastic.co/u/felixbarny)\
**Post date:** [June 24, 2020, 2:14pm UTC](https://discuss.elastic.co/t/no-jmx-metrics-in-jboss-eap-7-3-0/238438/2 "2020-06-24T14:14:10Z")

</div>

Hi and thanks for your question!

> [@sopgreg](#):
>
> `Found mbeans for object name jboss.as:subsystem=datasources,data-source=NomTDS_Pool,statistics=pool`

Maybe what's happening here is that JBoss creates a special `MBeanServer` where it registers the metrics to. Currently, we only track metrics from the platfrom MBean server (`ManagementFactory.getPlatformMBeanServer()`).

Could you try to find out if there are indeed multiple MBean servers by calling `MBeanServerFactory.findMBeanServer(null)`? If so, whats the domain (`MBeanServer#getDefaultDomain`) of the MBean server that contains the metrics you are looking for?

---

<div class="post-metadata">

**Author:** ![sopgreg](https://avatars.discourse-cdn.com/v4/letter/s/df788c/32.png) [@sopgreg](https://discuss.elastic.co/u/sopgreg)\
**Post date:** [June 24, 2020, 3:29pm UTC](https://discuss.elastic.co/t/no-jmx-metrics-in-jboss-eap-7-3-0/238438/3 "2020-06-24T15:29:23Z")

</div>

This is strange. There is only one MBean server, but querying the MBeans (returns more than 200), does not return the ones in question. Still, JConsole returns such MBeans and they can also be queried using a Jolokia servlet.

 ![grafik](https://us1.discourse-cdn.com/elastic/original/3X/c/7/c7f6688b718626c52d85fc3fb6eeda5c0e1655c1.png)  
 ![grafik](https://us1.discourse-cdn.com/elastic/original/3X/1/e/1e667fb23cc8dc73010e0bb625446f6f4fb18f83.png)

---

<div class="post-metadata">

**Author:** ![sopgreg](https://avatars.discourse-cdn.com/v4/letter/s/df788c/32.png) [@sopgreg](https://discuss.elastic.co/u/sopgreg)\
**Post date:** [June 24, 2020, 4:39pm UTC](https://discuss.elastic.co/t/no-jmx-metrics-in-jboss-eap-7-3-0/238438/4 "2020-06-24T16:39:32Z")

</div>

The MBean seems to be there:

 ![grafik](https://us1.discourse-cdn.com/elastic/original/3X/2/9/29a074f15e3efbd83fc94d67a5af9f2b52b92f91.png)

---

<div class="post-metadata">

**Author:** ![sopgreg](https://avatars.discourse-cdn.com/v4/letter/s/df788c/32.png) [@sopgreg](https://discuss.elastic.co/u/sopgreg)\
**Post date:** [June 25, 2020, 7:54am UTC](https://discuss.elastic.co/t/no-jmx-metrics-in-jboss-eap-7-3-0/238438/5 "2020-06-25T07:54:00Z")

</div>

I built a snapshot of the agent myself and put a breakpoint here:

`JmxMetricTracker.java:192`

Seems that no MBeans of domain "jboss:as" are caught here at all - maybe some kind of race condition/timing issue? Imho the listener is registered early enough.

But what's also interesting is, is that in `JmxMetricTracker#init` and via `ManagementFactory.getPlatformMBeanServer()`, this time the MBean server seems to be something different:

`org.jboss.as.jmx.PluggableMBeanServerImpl@6c89032e`

 ![grafik](https://us1.discourse-cdn.com/elastic/original/3X/b/6/b64a3e885620f4bc4b04e3ee233ce1f46fc57411.png)

---

<div class="post-metadata">

**Author:** ![felixbarny](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/felixbarny/32/27341_2.png) [@felixbarny](https://discuss.elastic.co/u/felixbarny)\
**Post date:** [June 25, 2020, 8:10am UTC](https://discuss.elastic.co/t/no-jmx-metrics-in-jboss-eap-7-3-0/238438/6 "2020-06-25T08:10:15Z")

</div>

That's interesting. Maybe JBoss overrides the default platform MBean server with a new one by calling `javax.management.MBeanServerFactory.createMBeanServer(null)`.

---

<div class="post-metadata">

**Author:** ![felixbarny](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/felixbarny/32/27341_2.png) [@felixbarny](https://discuss.elastic.co/u/felixbarny)\
**Post date:** [June 25, 2020, 9:13am UTC](https://discuss.elastic.co/t/no-jmx-metrics-in-jboss-eap-7-3-0/238438/7 "2020-06-25T09:13:02Z")

</div>

Actually, it seems like the platform MBean server can't be overridden once set.

> [@sopgreg](#):
>
> org.jboss.as.jmx.PluggableMBeanServerImpl@6c89032e

This seems to be the `toString()` representation whereas `PluggableMBeanServerImpl@4986` is the [object id](https://stackoverflow.com/questions/2322903/deciphering-variable-information-while-debugging-java). Could you check again if it's the same instance of the object?

What you could also try is to add the `capture_jmx_metrics` config after the server has already started. The agent will load the new configuration from `elasticapm.properties` every 30s.

---

<div class="post-metadata">

**Author:** ![sopgreg](https://avatars.discourse-cdn.com/v4/letter/s/df788c/32.png) [@sopgreg](https://discuss.elastic.co/u/sopgreg)\
**Post date:** [June 25, 2020, 9:52am UTC](https://discuss.elastic.co/t/no-jmx-metrics-in-jboss-eap-7-3-0/238438/8 "2020-06-25T09:52:06Z")

</div>

The MBean server seems to be the same during #init and later on once the server is started. This time, the name again changed to `JmxMBeanServer` instead of `PluggableMBeanServerImpl`

 ![grafik](https://us1.discourse-cdn.com/elastic/original/3X/9/3/9383fcc68a677de06fe11fcd330168f5a208abe5.png)

Adding `capture_jmx_metrics` at runtime finally worked:

```
2020-06-25 11:37:59,197 [elastic-apm-configuration-reloader] DEBUG co.elastic.apm.agent.jmx.JmxMetricTracker - Init JMX metric tracking with server org.jboss.as.jmx.PluggableMBeanServerImpl@bd63369
2020-06-25 11:37:59,978 [elastic-apm-configuration-reloader] DEBUG co.elastic.apm.agent.jmx.JmxMetricTracker - Found mbeans for object name jboss.as:subsystem=datasources,data-source=*,statistics=pool
2020-06-25 11:38:00,060 [elastic-apm-configuration-reloader] DEBUG co.elastic.apm.agent.jmx.JmxMetricTracker - Found number attribute AvailableCount=0
2020-06-25 11:38:00,252 [elastic-apm-configuration-reloader] DEBUG co.elastic.apm.agent.jmx.JmxMetricTracker - Found number attribute ActiveCount=0
2020-06-25 11:38:00,324 [elastic-apm-configuration-reloader] DEBUG co.elastic.apm.agent.jmx.JmxMetricTracker - Found number attribute InUseCount=0
2020-06-25 11:38:00,400 [elastic-apm-configuration-reloader] DEBUG co.elastic.apm.agent.jmx.JmxMetricTracker - Found number attribute TimedOut=0
2020-06-25 11:38:00,440 [elastic-apm-configuration-reloader] DEBUG co.elastic.apm.agent.jmx.JmxMetricTracker - Found number attribute WaitCount=0
2020-06-25 11:38:00,489 [elastic-apm-configuration-reloader] DEBUG co.elastic.apm.agent.jmx.JmxMetricTracker - Found number attribute AvailableCount=100
2020-06-25 11:38:00,497 [elastic-apm-configuration-reloader] DEBUG co.elastic.apm.agent.jmx.JmxMetricTracker - Found number attribute ActiveCount=10
2020-06-25 11:38:00,501 [elastic-apm-configuration-reloader] DEBUG co.elastic.apm.agent.jmx.JmxMetricTracker - Found number attribute InUseCount=0
2020-06-25 11:38:00,540 [elastic-apm-configuration-reloader] DEBUG co.elastic.apm.agent.jmx.JmxMetricTracker - Found number attribute TimedOut=0
```

---

<div class="post-metadata">

**Author:** ![felixbarny](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/felixbarny/32/27341_2.png) [@felixbarny](https://discuss.elastic.co/u/felixbarny)\
**Post date:** [June 25, 2020, 12:20pm UTC](https://discuss.elastic.co/t/no-jmx-metrics-in-jboss-eap-7-3-0/238438/9 "2020-06-25T12:20:47Z")

</div>

Ok, there seems to be something going on with the `NotificationListener` then. Could you try the [artifacts](https://apm-ci.elastic.co/blue/organizations/jenkins/apm-agent-java%2Fapm-agent-java-mbp/detail/PR-1253/1/artifacts) from this [PR](https://github.com/elastic/apm-agent-java/pull/1253) and set the log level to `trace` to test this out?

---

<div class="post-metadata">

**Author:** ![sopgreg](https://avatars.discourse-cdn.com/v4/letter/s/df788c/32.png) [@sopgreg](https://discuss.elastic.co/u/sopgreg)\
**Post date:** [June 25, 2020, 2:56pm UTC](https://discuss.elastic.co/t/no-jmx-metrics-in-jboss-eap-7-3-0/238438/10 "2020-06-25T14:56:53Z")

</div>

This is what we get. The agent gets notified of around 400 MBeans, but there are more than 3000 in JConsole. The "jboss:as" ones are not logged.

```
2: 2020-06-25 16:49:57,400 [main] INFO co.elastic.apm.agent.util.JmxUtils - Found JVM-specific OperatingSystemMXBean interface: com.sun.management.OperatingSystemMXBean

    246: 2020-06-25 16:49:59,006 [main] DEBUG co.elastic.apm.agent.configuration.StartupInfo - capture_jmx_metrics: 'object_name[jboss.as:subsystem=datasources,data-source=*,statistics=pool] attribute[AvailableCount] attribute[ActiveCount] attribute[InUseCount] attribute[TimedOut] attribute[WaitCount]' (source: C:\Server\apm/elasticapm.properties)

    277: 2020-06-25 16:50:00,694 [main] DEBUG co.elastic.apm.agent.jmx.JmxMetricTracker - Deferring initialization of JMX metric tracking until log manager is initialized

    279: 2020-06-25 16:50:01,853 [elastic-apm-jmx-init] DEBUG co.elastic.apm.agent.jmx.JmxMetricTracker - Init JMX metric tracking with server com.sun.jmx.mbeanserver.JmxMBeanServer@224ab400

    280: 2020-06-25 16:50:01,866 [elastic-apm-jmx-init] DEBUG co.elastic.apm.agent.jmx.JmxMetricTracker - Found no mbeans for object name jboss.as:subsystem=datasources,data-source=*,statistics=pool. Listening for MBeans added later.

    281: 2020-06-25 16:50:02,001 [main] TRACE co.elastic.apm.agent.jmx.JmxMetricTracker - Receiving MBean registration notification for jboss.modules:type=ModuleLoader,name=ModuleLoader-2

    282: 2020-06-25 16:50:02,001 [main] TRACE co.elastic.apm.agent.jmx.JmxMetricTracker - Receiving MBean registration notification for jboss.modules:type=ModuleLoader,name=LocalModuleLoader-3

    513: 2020-06-25 16:50:02,641 [main] TRACE co.elastic.apm.agent.jmx.JmxMetricTracker - Receiving MBean registration notification for jboss.threads:name="threadpool-1",type=thread-pool

    514: 2020-06-25 16:50:02,649 [main] TRACE co.elastic.apm.agent.jmx.JmxMetricTracker - Receiving MBean registration notification for jboss.msc:type=container,name=jboss-as

    515: 2020-06-25 16:50:02,835 [main] TRACE co.elastic.apm.agent.jmx.JmxMetricTracker - Receiving MBean registration notification for jboss.root:type=state

    516: 2020-06-25 16:50:02,895 [MSC service thread 1-3] TRACE co.elastic.apm.agent.jmx.JmxMetricTracker - Receiving MBean registration notification for jboss.modules:type=ModuleLoader,name=ServiceModuleLoader-4

    517: 2020-06-25 16:50:03,017 [MSC service thread 1-5] TRACE co.elastic.apm.agent.jmx.JmxMetricTracker - Receiving MBean registration notification for jboss.threads:name="threadpool-2",type=thread-pool

    659: 2020-06-25 16:50:13,420 [MSC service thread 1-3] TRACE co.elastic.apm.agent.jmx.JmxMetricTracker - Receiving MBean registration notification for jboss.threads:name="threadpool-3",type=thread-pool

    660: 2020-06-25 16:50:13,431 [MSC service thread 1-1] TRACE co.elastic.apm.agent.jmx.JmxMetricTracker - Receiving MBean registration notification for jboss.threads:name="threadpool-4",type=thread-pool

    661: 2020-06-25 16:50:13,514 [MSC service thread 1-8] TRACE co.elastic.apm.agent.jmx.JmxMetricTracker - Receiving MBean registration notification for org.xnio:type=Xnio,provider="nio"

    689: 2020-06-25 16:50:13,768 [MSC service thread 1-8] TRACE co.elastic.apm.agent.jmx.JmxMetricTracker - Receiving MBean registration notification for jboss.threads:name="management",type=thread-pool

    690: 2020-06-25 16:50:13,914 [MSC service thread 1-8] TRACE co.elastic.apm.agent.jmx.JmxMetricTracker - Receiving MBean registration notification for org.xnio:type=Xnio,provider="nio",worker="management"

    719: 2020-06-25 16:50:14,095 [MSC service thread 1-5] TRACE co.elastic.apm.agent.jmx.JmxMetricTracker - Receiving MBean registration notification for jboss.threads:name="default",type=thread-pool

    747: 2020-06-25 16:50:14,135 [MSC service thread 1-5] TRACE co.elastic.apm.agent.jmx.JmxMetricTracker - Receiving MBean registration notification for org.xnio:type=Xnio,provider="nio",worker="default"

    796: 2020-06-25 16:50:14,216 [MSC service thread 1-1] TRACE co.elastic.apm.agent.jmx.JmxMetricTracker - Receiving MBean registration notification for jboss.remoting.endpoint:name=Remoting-node_a-760439895

    861: 2020-06-25 16:50:14,278 [MSC service thread 1-1] TRACE co.elastic.apm.agent.jmx.JmxMetricTracker - Receiving MBean registration notification for jboss.remoting.handler:name=Remoting-node_a-remote-691247730

    862: 2020-06-25 16:50:14,279 [MSC service thread 1-1] TRACE co.elastic.apm.agent.jmx.JmxMetricTracker - Receiving MBean registration notification for jboss.remoting.handler:name=Remoting-node_a-remote+tls-282658987

    863: 2020-06-25 16:50:14,280 [MSC service thread 1-1] TRACE co.elastic.apm.agent.jmx.JmxMetricTracker - Receiving MBean registration notification for jboss.remoting.handler:name=Remoting-node_a-remoting-1331247756

    864: 2020-06-25 16:50:14,280 [MSC service thread 1-1] TRACE co.elastic.apm.agent.jmx.JmxMetricTracker - Receiving MBean registration notification for jboss.remoting.handler:name=Remoting-node_a-remote+http-136111724

    865: 2020-06-25 16:50:14,281 [MSC service thread 1-1] TRACE co.elastic.apm.agent.jmx.JmxMetricTracker - Receiving MBean registration notification for jboss.remoting.handler:name=Remoting-node_a-remote+https-896731320

    866: 2020-06-25 16:50:14,303 [MSC service thread 1-1] TRACE co.elastic.apm.agent.jmx.JmxMetricTracker - Receiving MBean registration notification for jboss.remoting.handler:name=Remoting-node_a-http-remoting-1843901870

    867: 2020-06-25 16:50:14,303 [MSC service thread 1-1] TRACE co.elastic.apm.agent.jmx.JmxMetricTracker - Receiving MBean registration notification for jboss.remoting.handler:name=Remoting-node_a-https-remoting-1548565077

    + 300 more
```

---

<div class="post-metadata">

**Author:** ![felixbarny](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/felixbarny/32/27341_2.png) [@felixbarny](https://discuss.elastic.co/u/felixbarny)\
**Post date:** [June 29, 2020, 7:51am UTC](https://discuss.elastic.co/t/no-jmx-metrics-in-jboss-eap-7-3-0/238438/11 "2020-06-29T07:51:49Z")

</div>

I have found the issue. Could you try out the build artifact from this PR: [https://github.com/elastic/apm-agent-java/pull/1258#issuecomment-650985749](https://github.com/elastic/apm-agent-java/pull/1258#issuecomment-650985749)

---

<div class="post-metadata">

**Author:** ![sopgreg](https://avatars.discourse-cdn.com/v4/letter/s/df788c/32.png) [@sopgreg](https://discuss.elastic.co/u/sopgreg)\
**Post date:** [June 29, 2020, 11:58am UTC](https://discuss.elastic.co/t/no-jmx-metrics-in-jboss-eap-7-3-0/238438/12 "2020-06-29T11:58:13Z")

</div>

While it does capture a lot more now, it still seems to miss the ",statistics=pool" beans:

 ![grafik](https://us1.discourse-cdn.com/elastic/original/3X/0/5/055d9c84ec0a78ab9f20ec405ef675f63b1ff15d.png)  
 ![grafik](https://us1.discourse-cdn.com/elastic/original/3X/4/7/478622f5c1531cf3319c7ea70fcc469f9293b4ef.png)

---

<div class="post-metadata">

**Author:** ![Sylvain\_Juge](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sylvain_juge/32/55521_2.png) [@Sylvain\_Juge](https://discuss.elastic.co/u/Sylvain_Juge)\
**Post date:** [July 1, 2020, 3:35pm UTC](https://discuss.elastic.co/t/no-jmx-metrics-in-jboss-eap-7-3-0/238438/13 "2020-07-01T15:35:04Z")

</div>

Hi @sopgreg, I've managed to reproduce the issue locally

- only when installing agent with `-javaagent` option
- MBean is not available on startup but registered at a later time.

---

<div class="post-metadata">

**Author:** ![Sylvain\_Juge](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sylvain_juge/32/55521_2.png) [@Sylvain\_Juge](https://discuss.elastic.co/u/Sylvain_Juge)\
**Post date:** [July 2, 2020, 1:08pm UTC](https://discuss.elastic.co/t/no-jmx-metrics-in-jboss-eap-7-3-0/238438/14 "2020-07-02T13:08:56Z")

</div>

We found a way to make it work, could you try [this snapshot](https://apm-ci.elastic.co/job/apm-agent-java/job/apm-agent-java-mbp/job/PR-1258/lastSuccessfulBuild/artifact/src/github.com/elastic/apm-agent-java/elastic-apm-agent/target/elastic-apm-agent-1.17.1-SNAPSHOT.jar) to check if it's now working as expected ?

The underlying issue is that Jboss does not seem to always trigger notifications for beans that are in the `boss.as` domain, but if you configure it with `jboss.as.expr` it should work. The fix in the snapshot just applies a fallback to detect when the bean is added, so you just can use the beans as you see them in `jconsole`.

---

<div class="post-metadata">

**Author:** ![sopgreg](https://avatars.discourse-cdn.com/v4/letter/s/df788c/32.png) [@sopgreg](https://discuss.elastic.co/u/sopgreg)\
**Post date:** [July 3, 2020, 6:10am UTC](https://discuss.elastic.co/t/no-jmx-metrics-in-jboss-eap-7-3-0/238438/15 "2020-07-03T06:10:40Z")

</div>

This fix/fallback solves the issue.

It's a bit weird what JBoss is doing here, as they recommend themselves to use the `jboss:as` MBean for the query (see [https://access.redhat.com/solutions/268793](https://access.redhat.com/solutions/268793)).

Thanks a lot for the efforts!

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 24, 2020, 2:15am UTC](https://discuss.elastic.co/t/no-jmx-metrics-in-jboss-eap-7-3-0/238438/16 "2020-07-24T02:15:17Z")

</div>

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.