Jolokia Module No key error

Elastic Stack Beats
1

Hi I'm trying to grab JVM metrics from

-mbean: 'mbean:type=FormatStatus,client=Client,format=Format,ftp=FtpId,FtpProtocol=sftp,FtpServer=server,FtpPort=22,dir=directory,filename=file.txt'
      attributes:
        - attr: LastError
          field: lasterror
          field_type: string
        - attr: LastFile
          field: lastfile
          field_type: string
        - attr: LastFileBlankLines
          field: lastfileblanklines
          field_type: integer
        - attr: LastFileError
          field: lastfileerror
          field_type: string
        - attr: LastFileFailedRecords
          field: lastfilefailedrecords
          field_type: integer
        - attr: LastFileLines
          field: lastfilelines
          field_type: integer
        - attr: LastFileLoadTimeMilliseconds
          field: lastfileloadtimemiliseconds
          field_type: integer
        - attr: LastFileLoadedRecords
          field: lastfileloadedrecords
          field_type: integer
        - attr: LastFileLoadedRecordsPerSecond
          field: lastfileloadedrecordspersecond
          field_type: integer
        - attr: Name
          field: name
          field_type: string
        - attr: Status
          field: status
          field_type: string

but it says no key found is there a way I can manually create the key? Does it mean that I need to add it to the template?

2

Can you run the Metricbeat 6.0.0 snapshot build with the jolokia.jmx debug selector enabled and post the log. I added some debug logging to the module that hasn't been released yet, so the snapshot build is required to get the extra info in the logs.

logging.level: debug
logging.selectors: ['jolokia.jmx`]

I'm pretty sure that key not found error indicates that one of the attributes you were expected did not exist in the response. You can have a look at the bean in JConsole to see its attributes.

3

Here is the results in the log. I'm not getting the key error any more but after refreshing the fields list they still aren't showing. I did get to see 2 of the metrics that had errors and commented those out for now.

2017-06-02T12:55:37-04:00 DBG [jolokia.jmx] The response body from jolokia host localhost:7777 is: [{"timestamp":1496422537,"status":200,"request":{"mbean":"java.lang:type=Runtime","attribute":"Uptime","type":"read"},"value":{"Uptime":88637288}},{"timestamp":1496422537,"status":200,"request":{"mbean":"java.lang:type=Memory","type":"read"},"value":{"Verbose":false,"ObjectPendingFinalizationCount":0,"NonHeapMemoryUsage":{"max":136314880,"committed":30081024,"init":24576000,"used":29053128},"HeapMemoryUsage":{"max":492830720,"committed":492830720,"init":33554432,"used":319881352},"ObjectName":{"objectName":"java.lang:type=Memory"}}},{"timestamp":1496422537,"status":200,"request":{"mbean":"java.lang:type=OperatingSystem","attribute":"ProcessCpuLoad","type":"read"},"value":{"ProcessCpuLoad":0.0}},{"timestamp":1496422537,"status":200,"request":{"mbean":"java.lang:type=ClassLoading","attribute":["LoadedClassCount","TotalLoadedClassCount"],"type":"read"},"value":{"LoadedClassCount":3843,"TotalLoadedClassCount":4108}},{"timestamp":1496422537,"status":200,"request":{"mbean":"java.lang:type=Threading","attribute":["DaemonThreadCount","PeakThreadCount","ThreadCount"],"type":"read"},"value":{"PeakThreadCount":38,"DaemonThreadCount":21,"ThreadCount":31}},{"timestamp":1496422537,"status":200,"request":{"mbean":"com.west.wic.campLoad:type=ServerStatus","attribute":["Status","Version"],"type":"read"},"value":{"Status":"Running","Version":"4.1.9 (20170331-0910 - radupa)"}},{"timestamp":1496422537,"status":200,"request":{"mbean":"com.west.wic.campLoad:FtpPort=22,FtpProtocol=sftp,FtpServer=127.0.0.1,client=client,dir=data_client_Seasonal_Allergy_inputs_Processed,filename=ALLERGY_REFILL_.MERGED.DUPLICATES,format=seasonalallergy,ftp=FtpId12,type=FormatStatus","attribute":["LastError","LastFile","LastFileBlankLines","LastFileError","LastFileFailedRecords","LastFileLines","LastFileLoadedRecords","Name","Status"],"type":"read"},"value":{"Name":"client - seasonalallergy","Status":"ready","LastFile":null,"LastFileLoadedRecords":0,"LastFileFailedRecords":0,"LastFileLines":0,"LastError":null,"LastFileBlankLines":0,"LastFileError":null}}]

4

I formatted the response JSON to make it easier to read.

[
  {
    "timestamp": 1496422537,
    "status": 200,
    "request": {
      "mbean": "java.lang:type=Runtime",
      "attribute": "Uptime",
      "type": "read"
    },
    "value": {
      "Uptime": 88637288
    }
  },
  {
    "timestamp": 1496422537,
    "status": 200,
    "request": {
      "mbean": "java.lang:type=Memory",
      "type": "read"
    },
    "value": {
      "Verbose": false,
      "ObjectPendingFinalizationCount": 0,
      "NonHeapMemoryUsage": {
        "max": 136314880,
        "committed": 30081024,
        "init": 24576000,
        "used": 29053128
      },
      "HeapMemoryUsage": {
        "max": 492830720,
        "committed": 492830720,
        "init": 33554432,
        "used": 319881352
      },
      "ObjectName": {
        "objectName": "java.lang:type=Memory"
      }
    }
  },
  {
    "timestamp": 1496422537,
    "status": 200,
    "request": {
      "mbean": "java.lang:type=OperatingSystem",
      "attribute": "ProcessCpuLoad",
      "type": "read"
    },
    "value": {
      "ProcessCpuLoad": 0
    }
  },
  {
    "timestamp": 1496422537,
    "status": 200,
    "request": {
      "mbean": "java.lang:type=ClassLoading",
      "attribute": [
        "LoadedClassCount",
        "TotalLoadedClassCount"
      ],
      "type": "read"
    },
    "value": {
      "LoadedClassCount": 3843,
      "TotalLoadedClassCount": 4108
    }
  },
  {
    "timestamp": 1496422537,
    "status": 200,
    "request": {
      "mbean": "java.lang:type=Threading",
      "attribute": [
        "DaemonThreadCount",
        "PeakThreadCount",
        "ThreadCount"
      ],
      "type": "read"
    },
    "value": {
      "PeakThreadCount": 38,
      "DaemonThreadCount": 21,
      "ThreadCount": 31
    }
  },
  {
    "timestamp": 1496422537,
    "status": 200,
    "request": {
      "mbean": "com.west.wic.campLoad:type=ServerStatus",
      "attribute": [
        "Status",
        "Version"
      ],
      "type": "read"
    },
    "value": {
      "Status": "Running",
      "Version": "4.1.9 (20170331-0910 - radupa)"
    }
  },
  {
    "timestamp": 1496422537,
    "status": 200,
    "request": {
      "mbean": "com.west.wic.campLoad:FtpPort=22,FtpProtocol=sftp,FtpServer=127.0.0.1,client=client,dir=dataclient_Seasonal_Allergy_inputs_Processed,filename=ALLERGY_REFILL__.MER,
      "attribute": [
        "LastError",
        "LastFile",
        "LastFileBlankLines",
        "LastFileError",
        "LastFileFailedRecords",
        "LastFileLines",
        "LastFileLoadedRecords",
        "Name",
        "Status"
      ],
      "type": "read"
    },
    "value": {
      "Name": "client - seasonalallergy",
      "Status": "ready",
      "LastFile": null,
      "LastFileLoadedRecords": 0,
      "LastFileFailedRecords": 0,
      "LastFileLines": 0,
      "LastError": null,
      "LastFileBlankLines": 0,
      "LastFileError": null
    }
  }
5

I did get 500 or so new fields with the new version of metricbeat. 1 of which conflicts haproxy.stat.check.health.last which was set to number but is now set to string. But I don't think that would be preventing the fields. I was wrong the message is showing this still

{ "message": "6 errors: metric key '' not found in response; metric key '' not found in response; metric key '' not found in response; metric key '' not found in response; metric key '' not found in response; metric key '' not found in response" }

6

I think null values are explicitly dropped from the object before being output by the Beat. So if it's the fields with null values that are missing then this would explain it.

7

I commented those out but no change.

2017-06-02T14:04:12-04:00 DBG [jolokia.jmx] The response body from jolokia host localhost:7777 is: [{"timestamp":1496426652,"status":200,"request":{"mbean":"java.lang:type=Runtime","attribute":"Uptime","type":"read"},"value":{"Uptime":92752901}},{"timestamp":1496426652,"status":200,"request":{"mbean":"java.lang:type=Memory","type":"read"},"value":{"Verbose":false,"ObjectPendingFinalizationCount":0,"NonHeapMemoryUsage":{"max":136314880,"committed":30212096,"init":24576000,"used":29155816},"HeapMemoryUsage":{"max":493355008,"committed":493355008,"init":33554432,"used":294691288},"ObjectName":{"objectName":"java.lang:type=Memory"}}},{"timestamp":1496426652,"status":200,"request":{"mbean":"java.lang:type=OperatingSystem","attribute":"ProcessCpuLoad","type":"read"},"value":{"ProcessCpuLoad":0.0}},{"timestamp":1496426652,"status":200,"request":{"mbean":"java.lang:type=ClassLoading","attribute":["LoadedClassCount","TotalLoadedClassCount"],"type":"read"},"value":{"LoadedClassCount":3843,"TotalLoadedClassCount":4108}},{"timestamp":1496426652,"status":200,"request":{"mbean":"java.lang:type=Threading","attribute":["DaemonThreadCount","PeakThreadCount","ThreadCount"],"type":"read"},"value":{"PeakThreadCount":38,"DaemonThreadCount":21,"ThreadCount":31}},{"timestamp":1496426652,"status":200,"request":{"mbean":"com.west.wic.campLoad:type=ServerStatus","attribute":["Status","Version"],"type":"read"},"value":{"Status":"Running","Version":"4.1.9 (20170331-0910 - radupa)"}},{"timestamp":1496426652,"status":200,"request":{"mbean":"com.west.wic.campLoad:FtpPort=22,FtpProtocol=sftp,FtpServer=127.0.0.1,client=client,dir=data_client_Seasonal_Allergy_inputs_Processed,filename=ALLERGY_REFILL_.MERGED.DUPLICATES,format=seasonalallergy,ftp=FtpId12,type=FormatStatus","attribute":["LastFileBlankLines","LastFileFailedRecords","LastFileLines","LastFileLoadedRecords","Name","Status"],"type":"read"},"value":{"com.west.wic.campLoad:FtpPort=22,FtpProtocol=sftp,FtpServer=127.0.0.1,client=client,dir=data_client_Seasonal_Allergy_inputs_Processed,filename=ALLERGY_REFILL_.MERGED.DUPLICATES,format=seasonalallergy,ftp=FtpId12,type=FormatStatus":{"Name":"client - seasonalallergy","Status":"ready","LastFileLoadedRecords":0,"LastFileFailedRecords":0,"LastFileLines":0,"LastFileBlankLines":0}}}]

it's missing all of these fields:
"Name": "client - seasonalallergy",
"Status": "ready",
"LastFile": null,
"LastFileLoadedRecords": 0,
"LastFileFailedRecords": 0,
"LastFileLines": 0,
"LastError": null,
"LastFileBlankLines": 0,
"LastFileError": null

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.