Metricbeat Jolokia Module Not Capturing all data

jason_0 · March 27, 2020, 5:26am

I have a RedHat AMQ setup with some dummy data in it that I'm trying to capture with Metricbeat. I'm able to capture standard MBean attributes such as java.lang:type=Runtime's 'Uptime'. I'm also able to capture some AMQ data using the following:

- module: jolokia
  service.name: "JOLOKIA"
  metricsets: ["jmx"]
  period: 10s
  hosts: ["10.1.43.40:8161"]
  namespace: "metrics"
  path: "console/jolokia/?ignoreErrors=true&canonicalNaming=false"
  username: "admin"
  password: "admin"
  http_method: 'POST'
  jmx.mappings:
    - mbean: 'java.lang:type=Runtime'
      attributes:
        - attr: Uptime
          field: uptime
    - mbean: 'org.apache.activemq.artemis:broker="0.0.0.0"'
      attributes:
        - attr: AddressMemoryUsage
          field: AddressMemoryUsage

However, I'm unable to get data from other MBeans example of the entire file is below. Note, the data I'm talking about is for MBean 'org.apache.activemq.artemis:broker="0.0.0.0",component=addresses,address="helloworld",subcomponent=queues,routing-type="anycast",queue="helloworld"'

- module: jolokia
  service.name: "JOLOKIA"
  metricsets: ["jmx"]
  period: 10s
  hosts: ["10.1.43.40:8161"]
  namespace: "metrics"
  path: "console/jolokia/?ignoreErrors=true&canonicalNaming=false"
  username: "admin"
  password: "admin"
  http_method: 'POST'
  jmx.mappings:
    - mbean: 'java.lang:type=Runtime'
      attributes:
        - attr: Uptime
          field: uptime
    - mbean: 'java.lang:type=Memory'
      attributes:
        - attr: HeapMemoryUsage
          field: memory.heap_usage
        - attr: NonHeapMemoryUsage
          field: memory.non_heap_usage
    - mbean: 'org.apache.activemq.artemis:broker="0.0.0.0"'
      attributes:
        - attr: AddressMemoryUsage
          field: AddressMemoryUsage
        - attr: ConnectionCount
          field: ConnectionCount
        - attr: TotalConnectionCount
          field: TotalConnectionCount
        - attr: TotalConsumerCount
          field: TotalConsumerCount
        - attr: TotalMessageCount
          field: TotalMessageCount
        - attr: TotalMessagesAcknowledged
          field: TotalMessagesAcknowledged
        - attr: TotalMessagesAdded
          field: TotalMessagesAdded
        - attr: Uptime
          field: BrokerUptime
    - mbean: 'org.apache.activemq.artemis:broker="0.0.0.0",component=addresses,address="helloworld",subcomponent=queues,routing-type="anycast",queue="helloworld"'
      attributes:
        - attr: MessageCount
          field: HelloWorldMessageCount

  jmx.application:
  jmx.instance:
  headers:
    Origin: http://10.1.43.40

I captured the logs and I can see the data is being provided in the response, but Metricbeat doesn't seem to understand it.

Mar 26 23:07:21 localhost metricbeat: 2020-03-26T23:07:20.995Z#011DEBUG#011[jolokia.jmx]#011jmx/config.go:499#011Jolokia response body#011{"host": "10.1.43.40:8161", "host": "10.1.43.40:8161", "uri": "http://10.1.43.40:8161/console/jolokia/%3FignoreErrors=true&canonicalNaming=false", "body": "[{\"request\":{\"mbean\":\"java.lang:type=Runtime\",\"attribute\":\"Uptime\",\"type\":\"read\"},\"value\":{\"Uptime\":12887343},\"timestamp\":1585264040,\"status\":200},{\"request\":{\"mbean\":\"java.lang:type=Memory\",\"attribute\":[\"HeapMemoryUsage\",\"NonHeapMemoryUsage\"],\"type\":\"read\"},\"value\":{\"HeapMemoryUsage\":{\"init\":536870912,\"committed\":536870912,\"max\":2147483648,\"used\":164582368},\"NonHeapMemoryUsage\":{\"init\":2555904,\"committed\":69230592,\"max\":-1,\"used\":67068048}},\"timestamp\":1585264040,\"status\":200},{\"request\":{\"mbean\":\"org.apache.activemq.artemis:broker=\\\"0.0.0.0\\\"\",\"attribute\":[\"AddressMemoryUsage\",\"ConnectionCount\",\"TotalConnectionCount\",\"TotalConsumerCount\",\"TotalMessageCount\",\"TotalMessagesAcknowledged\",\"TotalMessagesAdded\",\"Uptime\"],\"type\":\"read\"},\"value\":{\"AddressMemoryUsage\":111280,\"TotalMessageCount\":100,\"ConnectionCount\":0,\"Uptime\":\"18 hours 36 minutes\",\"TotalMessagesAdded\":100,\"TotalMessagesAcknowledged\":0,\"TotalConsumerCount\":0,\"TotalConnectionCount\":2},\"timestamp\":1585264040,\"status\":200},{\"request\":{\"mbean\":\"org.apache.activemq.artemis:address=\\\"helloworld\\\",broker=\\\"0.0.0.0\\\",component=addresses,queue=\\\"helloworld\\\",routing-type=\\\"anycast\\\",subcomponent=queues\",\"attribute\":\"MessageCount\",\"type\":\"read\"},\"value\":{\"MessageCount\":100},\"timestamp\":1585264040,\"status\":200}]", "type": "response"}
Mar 26 23:07:21 localhost metricbeat: 2020-03-26T23:07:20.995Z#011DEBUG#011[jolokia.jmx]#011jmx/data.go:205#011mapping: map[{mbean:java.lang:type=Memory attr:HeapMemoryUsage}:{Attr:HeapMemoryUsage Field:memory.heap_usage Event:} {mbean:java.lang:type=Memory attr:NonHeapMemoryUsage}:{Attr:NonHeapMemoryUsage Field:memory.non_heap_usage Event:} {mbean:java.lang:type=Runtime attr:Uptime}:{Attr:Uptime Field:uptime Event:} {mbean:org.apache.activemq.artemis:broker="0.0.0.0" attr:AddressMemoryUsage}:{Attr:AddressMemoryUsage Field:AddressMemoryUsage Event:} {mbean:org.apache.activemq.artemis:broker="0.0.0.0" attr:ConnectionCount}:{Attr:ConnectionCount Field:ConnectionCount Event:} {mbean:org.apache.activemq.artemis:broker="0.0.0.0" attr:TotalConnectionCount}:{Attr:TotalConnectionCount Field:TotalConnectionCount Event:} {mbean:org.apache.activemq.artemis:broker="0.0.0.0" attr:TotalConsumerCount}:{Attr:TotalConsumerCount Field:TotalConsumerCount Event:} {mbean:org.apache.activemq.artemis:broker="0.0.0.0" attr:TotalMessageCount}:{Attr:TotalMessageCount Field:TotalMessageCount Event:} {mbean:org.apache.activemq.artemis:broker="0.0.0.0" attr:TotalMessagesAcknowledged}:{Attr:TotalMessagesAcknowledged Field:TotalMessagesAcknowledged Event:} {mbean:org.apache.activemq.artemis:broker="0.0.0.0" attr:TotalMessagesAdded}:{Attr:TotalMessagesAdded Field:TotalMessagesAdded Event:} {mbean:org.apache.activemq.artemis:broker="0.0.0.0" attr:Uptime}:{Attr:Uptime Field:BrokerUptime Event:} {mbean:org.apache.activemq.artemis:broker="0.0.0.0",component=addresses,address="helloworld",subcomponent=queues,routing-type="anycast",queue="helloworld" attr:MessageCount}:{Attr:MessageCount Field:HelloWorldMessageCount Event:}]
Mar 26 23:07:21 localhost metricbeat: 2020-03-26T23:07:20.995Z#011DEBUG#011[processors]#011processing/processors.go:186#011Publish event: {
Mar 26 23:07:21 localhost metricbeat: "@timestamp": "2020-03-26T23:07:20.902Z",
Mar 26 23:07:21 localhost metricbeat: "@metadata": {
Mar 26 23:07:21 localhost metricbeat: "beat": "metricbeat",
Mar 26 23:07:21 localhost metricbeat: "type": "_doc",
Mar 26 23:07:21 localhost metricbeat: "version": "7.6.1"
Mar 26 23:07:21 localhost metricbeat: },
Mar 26 23:07:21 localhost metricbeat: "service": {
Mar 26 23:07:21 localhost metricbeat: "type": "jolokia",
Mar 26 23:07:21 localhost metricbeat: "address": "10.1.43.40:8161",
Mar 26 23:07:21 localhost metricbeat: "name": "JOLOKIA"
Mar 26 23:07:21 localhost metricbeat: },
Mar 26 23:07:21 localhost metricbeat: "jolokia": {
Mar 26 23:07:21 localhost metricbeat: "metrics": {
Mar 26 23:07:21 localhost metricbeat: "TotalMessagesAcknowledged": 0,
Mar 26 23:07:21 localhost metricbeat: "TotalConsumerCount": 0,
Mar 26 23:07:21 localhost metricbeat: "TotalConnectionCount": 2,
Mar 26 23:07:21 localhost metricbeat: "ConnectionCount": 0,
Mar 26 23:07:21 localhost metricbeat: "BrokerUptime": "18 hours 36 minutes",
Mar 26 23:07:21 localhost metricbeat: "memory": {
Mar 26 23:07:21 localhost metricbeat: "heap_usage": {
Mar 26 23:07:21 localhost metricbeat: "init": 5.36870912e+08,
Mar 26 23:07:21 localhost metricbeat: "committed": 5.36870912e+08,
Mar 26 23:07:21 localhost metricbeat: "max": 2.147483648e+09,
Mar 26 23:07:21 localhost metricbeat: "used": 1.64582368e+08
Mar 26 23:07:21 localhost metricbeat: },
Mar 26 23:07:21 localhost metricbeat: "non_heap_usage": {
Mar 26 23:07:21 localhost metricbeat: "committed": 6.9230592e+07,
Mar 26 23:07:21 localhost metricbeat: "max": -1,
Mar 26 23:07:21 localhost metricbeat: "used": 6.7068048e+07,
Mar 26 23:07:21 localhost metricbeat: "init": 2.555904e+06
Mar 26 23:07:21 localhost metricbeat: }
Mar 26 23:07:21 localhost metricbeat: },
Mar 26 23:07:21 localhost metricbeat: "AddressMemoryUsage": 111280,
Mar 26 23:07:21 localhost metricbeat: "TotalMessagesAdded": 100,
Mar 26 23:07:21 localhost metricbeat: "uptime": 1.2887343e+07,
Mar 26 23:07:21 localhost metricbeat: "TotalMessageCount": 100
Mar 26 23:07:21 localhost metricbeat: }
Mar 26 23:07:21 localhost metricbeat: },
Mar 26 23:07:21 localhost metricbeat: "ecs": {
Mar 26 23:07:21 localhost metricbeat: "version": "1.4.0"
Mar 26 23:07:21 localhost metricbeat: },
Mar 26 23:07:21 localhost metricbeat: "host": {
Mar 26 23:07:21 localhost metricbeat: "containerized": false,
Mar 26 23:07:21 localhost metricbeat: "name": "mq01",
Mar 26 23:07:21 localhost metricbeat: "hostname": "mq01",
Mar 26 23:07:21 localhost metricbeat: "architecture": "x86_64",

Can anyone help me to resolve this?

ChrsMark · March 30, 2020, 8:11am

Hi @jason_0!

This is weird indeed! Could you try to debug it with removing all the other fields(those that work) and keep just the problematic one?

@jsoriano could you verify that there is noting tricky missing here?

jsoriano · March 31, 2020, 7:42pm

Hey,

I think that the problem is that Metricbeat is not correctly parsing mbeans with multiple quoted properties, as happens here. It is parsing the value of broker property as "0.0.0.0",component=addresses,address="helloworld",subcomponent=queues,routing-type="anycast",queue="helloworld" instead of just "0.0.0.0". It takes as value everything from the first quote to the last one.

A possible workaround for this case can be to define the mbeans in the configuration with the properties in canonical order (alphabetical), in that case even when the mbean is not being correctly parsed, at least the canonicalized result should be the same. I.e. try this configuration (not tested):

...
    - mbean: 'org.apache.activemq.artemis:address="helloworld",broker="0.0.0.0",component=addresses,queue="helloworld",routing-type="anycast",subcomponent=queues'
      attributes:
        - attr: MessageCount
          field: HelloWorldMessageCount
...

I have opened an issue and a draft PR to describe and try to fix the problem:

Issue: https://github.com/elastic/beats/issues/17375
Draft PR: https://github.com/elastic/beats/issues/17374

jason_0 · March 31, 2020, 10:30pm

Thanks, @jsoriano & @ChrsMark. The workaround seems to work so that get's me moving again.

Thanks for submitting the PR, hopefully, we can get it merged in

system · April 28, 2020, 10:30pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.