(APM java-agent) JMX metrics does not show up in apm-* index

[joek (JEkelundh)]

Kibana version:
kibana-7.6.1-1.x86_64
Elasticsearch version:
elasticsearch-7.6.1-1.x86_64
APM Server version:
apm-server-7.6.1-1.x86_64
APM Agent language and version:
elastic-apm-agent-1.14.0.jar
Browser version:
Chrome any
Original install method (e.g. download page, yum, deb, from source, etc.) and version:
yum except for agent
Fresh install or upgraded from other version?
fresh insall
Is there anything special in your setup? For example, are you using the Logstash or Kafka outputs? Are you using a load balancer in front of the APM Servers? Have you changed index pattern, generated custom templates, changed agent configuration etc.
n/a
Description of the problem including expected versus actual behavior. Please include screenshots (if relevant):

Steps to reproduce:

1. Configure tomcat to use APM-agent - works fine - APM metrics are visible in Kibana
2. Configure some capture_jmx_metrics - does not show up in index

Config: (service name is correctly parsed from elasticapm.properties )
capture_jmx_metrics=object_name[com.zaxxer.hikari:type=Pool (HikariPool-*)] attribute[TotalConnections]
service_name=Tomcat7_APM_Agent
This works:
./check_jmx -U service:jmx:rmi:///jndi/rmi://logserver.je.local:2222/jmxrmi -O com.zaxxer.hikari:type="Pool (HikariPool-*)" -A TotalConnections
JMX OK - TotalConnections = 9 | 'TotalConnections'=9;;;

(Disabling std JMX did not make any difference)
Errors in browser console (if relevant):
n/a
Provide logs and/or server output (if relevant):
no jmx data shows up. JMX data verified with JConsole and misc cmdline tools
Only PS Scavenge and PS Marksweep visible in labels.name

[Eyal_Koren]

Hi and welcome to the forum.

If I understand correctly, the ObjectName's type property value is of this form: "Pool (HikariPool-*)" (meaning, the (HikariPool-*) is part of the type property and not name property).
Did you try configuring: capture_jmx_metrics=object_name[com.zaxxer.hikari:type="Pool (HikariPool-*)"] attribute[TotalConnections]?

If that doesn't work, please set log_level to DEBUG and upload the resulted log, maybe we can find hints there.

I hope this helps.

[joek (JEkelundh)]

Hi, Thanks for quick reply
I changed config as instructed and restarted in DEBUG mode.
Still no Hikari-stuff visible in Kibana.
Could not attach logfile so I attach some lines:

egrep -i "jmx" /usr/share/tomcat/apmagent.log

2020-03-18 10:53:27.111 [main] INFO co.elastic.apm.agent.util.JmxUtils - Found JVM-specific OperatingSystemMXBean interface: com.sun.management.OperatingSystemMXBean
2020-03-18 10:53:27.424 [main] DEBUG co.elastic.apm.agent.configuration.StartupInfo - capture_jmx_metrics: 'object_name[com.zaxxer.hikari:type="Pool (HikariPool-*)"] attribute[TotalConnections]' (source: /opt/apm-agent/lib/elasticapm.properties)
2020-03-18 10:53:27.582 [main] DEBUG co.elastic.apm.agent.jmx.JmxMetricTracker - Init JMX metric tracking with server com.sun.jmx.mbeanserver.JmxMBeanServer@1376c05c
2020-03-18 10:53:27.585 [main] DEBUG co.elastic.apm.agent.jmx.JmxMetricTracker - Found mbeans for object name com.zaxxer.hikari:type="Pool (HikariPool-*)"

Other lines:
2020-03-18 10:53:46.739 [http-bio-8080-exec-2] DEBUG co.elastic.apm.agent.bci.ElasticApmAgent - Method match for instrumentation ExecuteBatchInstrumentation: (((name(equals(executeBatch)) or name(equals(executeLargeBatch))) and hasParameter(ofSize(0))) and isPublic()) matches public long[] com.zaxxer.hikari.pool.ProxyStatement.executeLargeBatch() throws java.sql.SQLException

2020-03-18 10:53:52.213 [HikariPool-1 connection adder] DEBUG co.elastic.apm.agent.impl.transaction.AbstractSpan - increment references to '' 00-d2bbd04e712c3144c09b9fefdff3b445-4cec43b361033532-01 (220d35d) (4)
2020-03-18 10:53:52.214 [HikariPool-1 connection adder] DEBUG co.elastic.apm.agent.impl.ElasticApmTracer - Activating '' 00-d2bbd04e712c3144c09b9fefdff3b445-4cec43b361033532-01 (220d35d) on thread 68
2020-03-18 10:53:52.238 [HikariPool-1 connection adder] DEBUG co.elastic.apm.agent.impl.transaction.AbstractSpan - increment references to '' 00-d2bbd04e712c3144c09b9fefdff3b445-4cec43b361033532-01 (220d35d) (8)

[joek (JEkelundh)]

Hi,
Reverted to my original config and extended it a bit and from agent it looks OK (?)

At least it says:
MBean added at runtime: com.zaxxer.hikari:type=Pool (HikariPool-*)
Found mbeans for object name com.zaxxer.hikari:type=Pool (HikariPool-*)
Found number attribute TotalConnections=1
Found number attribute ActiveConnections=0
Found number attribute IdleConnections=1
Found number attribute ThreadsAwaitingConnection=0

Registering JMX metric com.zaxxer.hikari:type=Pool (HikariPool-1) TotalConnections.null as metric_name: jvm.jmx.TotalConnections labels: type=Pool (HikariPool-1)
Registering JMX metric com.zaxxer.hikari:type=Pool (HikariPool-1) ActiveConnections.null as metric_name: jvm.jmx.ActiveConnections labels: type=Pool (HikariPool-1)

and so on ....

But still nothing visible in Kibana (std APM metrics gets delivered)

[Eyal_Koren]

Yes, this looks OK.
Custom metrics would not show up automatically in Kibana, but they are stored in Elasticsearch as explained in documentation and you can create a custom visualization for them.

[joek (JEkelundh)]

Hi,
OK, then I'll follow the Kibana trail.
My final config for reference:
capture_jmx_metrics=object_name[com.zaxxer.hikari:type=Pool (HikariPool-*)] attribute[TotalConnections] attribute[ActiveConnections] attribute[IdleConnections] attribute[ThreadsAwaitingConnection]

Thanks for guiding !