# Jolokia jmx Mbean collection

**URL:** <https://discuss.elastic.co/t/jolokia-jmx-mbean-collection/91442>\
**Category:** Beats\
**Tags:** metricbeat\
**Created:** [June 30, 2017, 1:51pm UTC](https://discuss.elastic.co/t/jolokia-jmx-mbean-collection/91442 "2017-06-30T13:51:51Z")\
**Posts on this page:** 12\
**Page:** 1

<div class="post-metadata">

**Author:** ![flowe](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/flowe/32/19665_2.png) [@flowe](https://discuss.elastic.co/u/flowe)\
**Post date:** [June 30, 2017, 1:51pm UTC](https://discuss.elastic.co/t/jolokia-jmx-mbean-collection/91442/1 "2017-06-30T13:51:51Z")

</div>

Hi there,

I'm trying to parse jmx mbean properties using jolokia module with metricbeat but have problems when having multiple datasource mbean entries (no problem when having one unique entry).

metricbeat 5.4.3  
logstash:5.4.2  
elasticsearch:5.4.2  
jolokia-jvm-1.3.6-agent.jar

I tried different synthaxes (double quoted, escaping chars or not, etc.) but either got **No key error** or **Unmarshall error** or **\_grokparsefailure** tag in output and not any new fields in my index.

Nothing in the metricbeat logs though :

`fetches.jolokia-jmx.events=30 fetches.jolokia-jmx.success=30`

Thanks in advance for any help!

- jolokia agent json list extract :

> ```
> {
> "timestamp": 1498825349,
> "status": 200,
> "request": {
> "type": "list"
> },
> "value": {
> ...
> "Catalina": {
> ...
> "class=javax.sql.DataSource,context=/akuiteo,host=localhost,name=\"jdbc/V9GestionDS\",type=DataSource": {
> "desc": null,
> "class": "org.apache.tomcat.util.modeler.BaseModelMBean",
> "attr": {
> "maxIdle": {
> "desc": "Introspected attribute maxIdle",
> "type": "int",
> "rw": true
> },
> "maxActive": {
> "desc": "Introspected attribute maxActive",
> "type": "int",
> "rw": true
> }
> }
> ...
> },
> "class=javax.sql.DataSource,context=/akuiteo.collabs,host=localhost,name=\"jdbc/V9GestionDS\",type=DataSource": {
> ...
> }
> }
> }
> }
> 
> ```

- metricbeat.yml extract :

> ```
> - module: jolokia
> metricsets: ["jmx"]
> enabled: true
> period: 1s
> hosts: ["localhost:8778"]
> namespace: "metrics"
> jmx.mappings:
> - mbean: 'java.lang:type=Runtime'
> attributes:
> - attr: Uptime
> field: uptime
> - mbean: 'Catalina:type=Server'
> attributes:
> - attr: serverInfo
> field: tomcat_server_info
> - mbean: 'Catalina:class=javax.sql.DataSource,context=\/akuiteo,host=localhost,name=jdbc\/V9GestionDS,type=DataSource'
> attributes:
> - attr: maxActive
> field: datasource_max_active
> - attr: numActive
> field: datasource_max_active
> - attr: numIdle
> field: datasource_num_idle
> 
> ```

---

<div class="post-metadata">

**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\
**Post date:** [July 4, 2017, 7:16am UTC](https://discuss.elastic.co/t/jolokia-jmx-mbean-collection/91442/2 "2017-07-04T07:16:44Z")

</div>

So your Setup is MB -\> LS -\> ES and you do some parsing on the LS side? How does your LS config look like?

Could you output to file and see wha the format created by metricbeat is?

---

<div class="post-metadata">

**Author:** ![flowe](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/flowe/32/19665_2.png) [@flowe](https://discuss.elastic.co/u/flowe)\
**Post date:** [July 4, 2017, 8:35am UTC](https://discuss.elastic.co/t/jolokia-jmx-mbean-collection/91442/3 "2017-07-04T08:35:28Z")

</div>

I'm not doing any parsing with logstash on metricbeat output yet.  
My problem appears during metricbeat trying to parse a jmx entries array exposed by Jolokia agent :

> Cannot unmarshal json response: json: cannot unmarshal object into Go value of type jmx.Entry

Is there a syntax to do that ?

- Logstash pipeline conf (only parsing filebeat output expected) :

> ```
> filter {
> if [@metadata][beat] == "filebeat" {
> grok {
> patterns_dir => "/usr/share/logstash/pipeline/patterns/"
> match => { "message" => "%{TIMESTAMP_ISO8601:timestamp}\|%{LOGLEVEL:log-level}%{SPACE}\|%{WORD:user};%{DATE_FR:startDate};%{DATE_FR:endDate};%{JAVA_CALL:serviceCall};%{NUMBER:serviceTime:int};%{NUMBER:databaseTime:int};" }
> }
> mutate {
> convert => { "serviceTime" => "integer"
> "dataBaseTime" => "integer"
> }
> }
> }
> }
> 
> ```

> ```
> output {
> elasticsearch {
> hosts => ["elasticsearch:9200"]
> index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
> document_type => "%{[@metadata][type]}"
> }
> }
> 
> ```

- Metricbeat debug log output with 1 syntax :

> ```
> 2017-07-04T10:14:19+02:00 DBG Publish: {
> "@timestamp": "2017-07-04T08:14:19.433Z",
> "beat": {
> "hostname": "machinename",
> "name": "machinename",
> "version": "5.4.3"
> },
> "error": "Cannot unmarshal json response: json: cannot unmarshal object into Go value of type []jmx.Entry",
> "jolokia": {
> "jmx": {}
> },
> "metricset": {
> "host": "localhost:8777",
> "module": "jolokia",
> "name": "jmx",
> "rtt": 2003
> },
> "type": "metricsets"
> }
> 
> ```

- Metricbeat debug log output with another syntax :

> ```
> 2017-07-04T10:28:16+02:00 DBG Publish: {
> "@timestamp": "2017-07-04T08:28:16.987Z",
> "beat": {
> "hostname": "machinename",
> "name": "machinename",
> "version": "5.4.3"
> },
> "jolokia": {
> "datasource": {}
> },
> "metricset": {
> "host": "localhost:8777",
> "module": "jolokia",
> "name": "jmx",
> "namespace": "datasource",
> "rtt": 10992
> },
> "type": "metricsets"
> }
> 
> ```

Thanks for your help.

---

<div class="post-metadata">

**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\
**Post date:** [July 6, 2017, 2:08pm UTC](https://discuss.elastic.co/t/jolokia-jmx-mbean-collection/91442/4 "2017-07-06T14:08:30Z")

</div>

Ah, I see. It seems your json output is very different from what we expect: [https://github.com/elastic/beats/blob/master/metricbeat/module/jolokia/jmx/data.go#L18](https://github.com/elastic/beats/blob/master/metricbeat/module/jolokia/jmx/data.go#L18)

Do you have any idea why these 2 outputs could be so different? But platform are you jolokia running on? It looks like Tomcat. If yes, which version?

---

<div class="post-metadata">

**Author:** ![flowe](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/flowe/32/19665_2.png) [@flowe](https://discuss.elastic.co/u/flowe)\
**Post date:** [July 7, 2017, 8:07am UTC](https://discuss.elastic.co/t/jolokia-jmx-mbean-collection/91442/5 "2017-07-07T08:07:01Z")

</div>

It is Tomcat 7.0.41 with one common context (and one resource Oracle Datasource for that context) to three deployed web apps (It might be why there are 3 Datasources entries in Mbean collection).

I can't really read go syntax unfortunately.  
But the output I get for datasources is close enough to, for example java.lang GC metrics (two entries with same type GarbageCollector but different names) which are parsed right.

- Extract of jolokia agent json output with java.lang GC (OK) and Catalina Datasource (NOK) :

> ```
> "request": {
> "type": "list"
> },
> "status": 200,
> "timestamp": 1499353927,
> "value": {
> "Catalina": {
> "class=javax.sql.DataSource,context=/akuiteo,host=localhost,name=\"jdbc/V9GestionDS\",type=DataSource": {
> "attr": {
> "accessToUnderlyingConnectionAllowed": {
> "desc": "Introspected attribute accessToUnderlyingConnectionAllowed",
> "rw": true,
> "type": "boolean"
> }
> },
> "class": "org.apache.tomcat.util.modeler.BaseModelMBean",
> "desc": null
> },
> "class=... ,context=/akuiteo.clients,host=...,name=\"jdbc/V9GestionDS\",type=DataSource": {
> "attr": {
> "accessToUnderlyingConnectionAllowed": {
> "desc": "Introspected attribute accessToUnderlyingConnectionAllowed",
> "rw": true,
> "type": "boolean"
> }
> }
> }
> }
> "java.lang": {
> "name=PS MarkSweep,type=GarbageCollector": {
> "attr": {
> "CollectionCount": {
> "desc": "CollectionCount",
> "rw": false,
> "type": "long"
> },
> "CollectionTime": {
> "desc": "CollectionTime",
> "rw": false,
> "type": "long"
> }
> },
> "class": "sun.management.GarbageCollectorImpl",
> "desc": "Information on the management interface of the MBean"
> },
> "name=PS Scavenge,type=GarbageCollector": {
> "attr": {
> "CollectionCount": {
> "desc": "CollectionCount",
> "rw": false,
> "type": "long"
> },
> "CollectionTime": {
> "desc": "CollectionTime",
> "rw": false,
> "type": "long"
> }
> },
> "class": "sun.management.GarbageCollectorImpl",
> "desc": "Information on the management interface of the MBean"
> }
> }
> }
> 
> ```

---

<div class="post-metadata">

**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\
**Post date:** [July 10, 2017, 12:45pm UTC](https://discuss.elastic.co/t/jolokia-jmx-mbean-collection/91442/6 "2017-07-10T12:45:19Z")

</div>

Interesting. I think you are right with your assumption. The current implementation does not support lists. It event states in the response object:

```auto
"request": {
  "type": "list"
},

```

If I look at our test example there is always `"type": "read"`.

Could you open a feature request for this on Github? Please provide there if possible a config with the direct jolokia output for this config. This will make testing and implementing the change much easier.

---

<div class="post-metadata">

**Author:** ![flowe](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/flowe/32/19665_2.png) [@flowe](https://discuss.elastic.co/u/flowe)\
**Post date:** [July 12, 2017, 1:16pm UTC](https://discuss.elastic.co/t/jolokia-jmx-mbean-collection/91442/7 "2017-07-12T13:16:39Z")

</div>

That might be a problem on jolokia agent side. I should be able to get the Datasource attributes using a "read" GET request (instead of list), in order to be compliant with jolokia metricbeat module actual functionning.  
Unless you consider that could be a interesting new feature, I'll try first to ask about the possibility to do so on jolokia github page before opening a feature request here (which could be another solution to that problem).  
Thanks again.

---

<div class="post-metadata">

**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\
**Post date:** [July 13, 2017, 9:59am UTC](https://discuss.elastic.co/t/jolokia-jmx-mbean-collection/91442/8 "2017-07-13T09:59:49Z")

</div>

SGTM. Could you link the Github issue here? I'm interested to learn more on what these configs options are or not. If it's a default setting, would probably be good for metricbeat do "understand" it.

---

<div class="post-metadata">

**Author:** ![flowe](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/flowe/32/19665_2.png) [@flowe](https://discuss.elastic.co/u/flowe)\
**Post date:** [July 13, 2017, 10:45am UTC](https://discuss.elastic.co/t/jolokia-jmx-mbean-collection/91442/9 "2017-07-13T10:45:09Z")

</div>

> <https://github.com/rhuss/jolokia/issues/330>

---

<div class="post-metadata">

**Author:** ![flowe](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/flowe/32/19665_2.png) [@flowe](https://discuss.elastic.co/u/flowe)\
**Post date:** [July 17, 2017, 1:50pm UTC](https://discuss.elastic.co/t/jolokia-jmx-mbean-collection/91442/10 "2017-07-17T13:50:47Z")

</div>

The problem was on Tomcat side (BasicDataSource does not implement a loginTimeout function), which caused a exception reading the DataSource JMX bean tree.

A simple workaround is to add "ignoreErrors=true" into jolokia module path field in metricbeat config file, so that even there is an error reading a part of Jmx bean, only the implicated field will not be readable.

The correct syntax below.

Thanks again.

metricbeat.yml extract :

```
- module: jolokia
  metricsets: ["jmx"]
  enabled: true
  period: 1s
  hosts: ["localhost:8778"]
  path: "/jolokia/?ignoreErrors=true&canonicalNaming=false"
  namespace: "metrics"
  jmx.mappings:
    - mbean: 'Catalina:class=javax.sql.DataSource,context=/akuiteo,host=localhost,name="jdbc/V9GestionDS",type=DataSource'
      attributes:
        - attr: maxActive
          field: datasource_max_active
        - attr: numActive
          field: datasource_max_active
        - attr: numIdle
          field: datasource_num_idle
```

---

<div class="post-metadata">

**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\
**Post date:** [July 17, 2017, 2:45pm UTC](https://discuss.elastic.co/t/jolokia-jmx-mbean-collection/91442/11 "2017-07-17T14:45:00Z")

</div>

Great to hear you got it working. Thanks a lot for sharing the solution. Wasn't aware of the ignoreErrors flag.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [August 14, 2017, 2:45pm UTC](https://discuss.elastic.co/t/jolokia-jmx-mbean-collection/91442/12 "2017-08-14T14:45:02Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.