Jolokia jmx Mbean collection

flowe · June 30, 2017, 1:51pm

Hi there,

I'm trying to parse jmx mbean properties using jolokia module with metricbeat but have problems when having multiple datasource mbean entries (no problem when having one unique entry).

metricbeat 5.4.3
logstash:5.4.2
elasticsearch:5.4.2
jolokia-jvm-1.3.6-agent.jar

I tried different synthaxes (double quoted, escaping chars or not, etc.) but either got No key error or Unmarshall error or _grokparsefailure tag in output and not any new fields in my index.

Nothing in the metricbeat logs though :

fetches.jolokia-jmx.events=30 fetches.jolokia-jmx.success=30

Thanks in advance for any help!

jolokia agent json list extract :

{
  "timestamp": 1498825349,
  "status": 200,
  "request": {
    "type": "list"
  },
  "value": {
  ...
    "Catalina": {
    ...
       "class=javax.sql.DataSource,context=/akuiteo,host=localhost,name=\"jdbc/V9GestionDS\",type=DataSource": {
       "desc": null,
       "class": "org.apache.tomcat.util.modeler.BaseModelMBean",
       "attr": {
         "maxIdle": {
           "desc": "Introspected attribute maxIdle",
           "type": "int",
           "rw": true
         },
         "maxActive": {
           "desc": "Introspected attribute maxActive",
           "type": "int",
           "rw": true
         }
       }
       ...
      },
      "class=javax.sql.DataSource,context=/akuiteo.collabs,host=localhost,name=\"jdbc/V9GestionDS\",type=DataSource": {
      ...
      }
    }
  }
}

metricbeat.yml extract :

- module: jolokia
  metricsets: ["jmx"]
  enabled: true
  period: 1s
  hosts: ["localhost:8778"]
  namespace: "metrics"
  jmx.mappings:
    - mbean: 'java.lang:type=Runtime'
      attributes:
        - attr: Uptime
          field: uptime
    - mbean: 'Catalina:type=Server'
      attributes:
        - attr: serverInfo
          field: tomcat_server_info
    - mbean: 'Catalina:class=javax.sql.DataSource,context=\/akuiteo,host=localhost,name=jdbc\/V9GestionDS,type=DataSource'
      attributes:
        - attr: maxActive
          field: datasource_max_active
        - attr: numActive
          field: datasource_max_active
        - attr: numIdle
          field: datasource_num_idle

ruflin · July 4, 2017, 7:16am

So your Setup is MB -> LS -> ES and you do some parsing on the LS side? How does your LS config look like?

Could you output to file and see wha the format created by metricbeat is?

flowe · July 4, 2017, 8:35am

I'm not doing any parsing with logstash on metricbeat output yet.
My problem appears during metricbeat trying to parse a jmx entries array exposed by Jolokia agent :

Cannot unmarshal json response: json: cannot unmarshal object into Go value of type jmx.Entry

Is there a syntax to do that ?

Logstash pipeline conf (only parsing filebeat output expected) :

filter {
  if [@metadata][beat] == "filebeat" {
     grok {
       patterns_dir => "/usr/share/logstash/pipeline/patterns/"
       match => { "message" => "%{TIMESTAMP_ISO8601:timestamp}\|%{LOGLEVEL:log-level}%{SPACE}\|%{WORD:user};%{DATE_FR:startDate};%{DATE_FR:endDate};%{JAVA_CALL:serviceCall};%{NUMBER:serviceTime:int};%{NUMBER:databaseTime:int};" }
     }
    mutate {
      convert => { "serviceTime" => "integer"
        "dataBaseTime" => "integer"
      }
    }
  }
}

output {
  elasticsearch {
    hosts => ["elasticsearch:9200"]
    index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
    document_type => "%{[@metadata][type]}"
  }
}

Metricbeat debug log output with 1 syntax :

2017-07-04T10:14:19+02:00 DBG  Publish: {
  "@timestamp": "2017-07-04T08:14:19.433Z",
  "beat": {
    "hostname": "machinename",
    "name": "machinename",
    "version": "5.4.3"
  },
  "error": "Cannot unmarshal json response: json: cannot unmarshal object into Go value of type []jmx.Entry",
  "jolokia": {
    "jmx": {}
  },
  "metricset": {
    "host": "localhost:8777",
    "module": "jolokia",
    "name": "jmx",
    "rtt": 2003
  },
  "type": "metricsets"
}

Metricbeat debug log output with another syntax :

2017-07-04T10:28:16+02:00 DBG  Publish: {
  "@timestamp": "2017-07-04T08:28:16.987Z",
    "beat": {
    "hostname": "machinename",
    "name": "machinename",
     "version": "5.4.3"
  },
  "jolokia": {
    "datasource": {}
  },
  "metricset": {
    "host": "localhost:8777",
    "module": "jolokia",
    "name": "jmx",
    "namespace": "datasource",
    "rtt": 10992
  },
  "type": "metricsets"
}

Thanks for your help.

ruflin · July 6, 2017, 2:08pm

Ah, I see. It seems your json output is very different from what we expect: https://github.com/elastic/beats/blob/master/metricbeat/module/jolokia/jmx/data.go#L18

Do you have any idea why these 2 outputs could be so different? But platform are you jolokia running on? It looks like Tomcat. If yes, which version?

flowe · July 7, 2017, 8:07am

It is Tomcat 7.0.41 with one common context (and one resource Oracle Datasource for that context) to three deployed web apps (It might be why there are 3 Datasources entries in Mbean collection).

I can't really read go syntax unfortunately.
But the output I get for datasources is close enough to, for example java.lang GC metrics (two entries with same type GarbageCollector but different names) which are parsed right.

Extract of jolokia agent json output with java.lang GC (OK) and Catalina Datasource (NOK) :

"request": {
  "type": "list"
},
"status": 200,
"timestamp": 1499353927,
"value": {
  "Catalina": {
    "class=javax.sql.DataSource,context=/akuiteo,host=localhost,name=\"jdbc/V9GestionDS\",type=DataSource": {
      "attr": {
        "accessToUnderlyingConnectionAllowed": {
          "desc": "Introspected attribute accessToUnderlyingConnectionAllowed",
          "rw": true,
          "type": "boolean"
        }
      },
      "class": "org.apache.tomcat.util.modeler.BaseModelMBean",
      "desc": null
    },
    "class=... ,context=/akuiteo.clients,host=...,name=\"jdbc/V9GestionDS\",type=DataSource": {
      "attr": {
        "accessToUnderlyingConnectionAllowed": {
          "desc": "Introspected attribute accessToUnderlyingConnectionAllowed",
          "rw": true,
          "type": "boolean"
         }
       }
     }
  }
  "java.lang": {
     "name=PS MarkSweep,type=GarbageCollector": {
       "attr": {
         "CollectionCount": {
           "desc": "CollectionCount",
           "rw": false,
           "type": "long"
         },
         "CollectionTime": {
           "desc": "CollectionTime",
           "rw": false,
           "type": "long"
          }
       },
       "class": "sun.management.GarbageCollectorImpl",
       "desc": "Information on the management interface of the MBean"
      },
      "name=PS Scavenge,type=GarbageCollector": {
        "attr": {
          "CollectionCount": {
            "desc": "CollectionCount",
            "rw": false,
            "type": "long"
          },
          "CollectionTime": {
            "desc": "CollectionTime",
            "rw": false,
            "type": "long"
          }
        },
        "class": "sun.management.GarbageCollectorImpl",
        "desc": "Information on the management interface of the MBean"
      }
    }
  }

ruflin · July 10, 2017, 12:45pm

Interesting. I think you are right with your assumption. The current implementation does not support lists. It event states in the response object:

"request": {
  "type": "list"
},

If I look at our test example there is always "type": "read".

Could you open a feature request for this on Github? Please provide there if possible a config with the direct jolokia output for this config. This will make testing and implementing the change much easier.

flowe · July 12, 2017, 1:16pm

That might be a problem on jolokia agent side. I should be able to get the Datasource attributes using a "read" GET request (instead of list), in order to be compliant with jolokia metricbeat module actual functionning.
Unless you consider that could be a interesting new feature, I'll try first to ask about the possibility to do so on jolokia github page before opening a feature request here (which could be another solution to that problem).
Thanks again.

ruflin · July 13, 2017, 9:59am

SGTM. Could you link the Github issue here? I'm interested to learn more on what these configs options are or not. If it's a default setting, would probably be good for metricbeat do "understand" it.

flowe · July 13, 2017, 10:45am

flowe · July 17, 2017, 1:50pm

The problem was on Tomcat side (BasicDataSource does not implement a loginTimeout function), which caused a exception reading the DataSource JMX bean tree.

A simple workaround is to add "ignoreErrors=true" into jolokia module path field in metricbeat config file, so that even there is an error reading a part of Jmx bean, only the implicated field will not be readable.

The correct syntax below.

Thanks again.

metricbeat.yml extract :

- module: jolokia
  metricsets: ["jmx"]
  enabled: true
  period: 1s
  hosts: ["localhost:8778"]
  path: "/jolokia/?ignoreErrors=true&canonicalNaming=false"
  namespace: "metrics"
  jmx.mappings:
    - mbean: 'Catalina:class=javax.sql.DataSource,context=/akuiteo,host=localhost,name="jdbc/V9GestionDS",type=DataSource'
      attributes:
        - attr: maxActive
          field: datasource_max_active
        - attr: numActive
          field: datasource_max_active
        - attr: numIdle
          field: datasource_num_idle

ruflin · July 17, 2017, 2:45pm

Great to hear you got it working. Thanks a lot for sharing the solution. Wasn't aware of the ignoreErrors flag.

system · August 14, 2017, 2:45pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.