Jolokia module not displaying threadcount in kibana

Elastic Stack Beats
,
1

Like the title says I cannot see the amount of threads in kibana here is my current jolokia.yml file:

# Module: jolokia
# Docs: https://www.elastic.co/guide/en/beats/metricbeat/7.1/metricbeat-module-jolokia.html

- module: jolokia
  metricsets: ["jmx"]
  period: 10s
  hosts: ["takenout"]
  namespace: "metrics"
  path: "/jolokia/?ignoreErrors=true&canonicalNaming=false"
  #username: "user"
  #password: "secret"
  jmx.mappings:
    - mbean: 'java.lang:type=Runtime'
      attributes:
        - attr: Uptime
          field: uptime
    - mbean: 'java.lang:type=Threading'
      attributes:
        - attr: ThreadCount
#       field: threading.thread_count
#    -mbean: 'java.lang:type=Memory'
#      attributes:
#         - attr: HeapMemoryUsage
#          field: memory.heap_usage
#        - attr: NonHeapMemoryUsage
#          field: memory.non_heap_usage
    # GC Metrics - this depends on what is available on your JVM
    #- mbean: 'java.lang:type=GarbageCollector,name=ConcurrentMarkSweep'
    #  attributes:
    #    - attr: CollectionTime
    #      field: gc.cms_collection_time
    #    - attr: CollectionCount
    #      field: gc.cms_collection_count

  jmx.application:
  jmx.instance:
2

Hi @cjb312,

At a first glance I see that the field name for the thread count is commented out, can you try to remove the comment for the line with field: threading.thread_count?

If this doesn't fix the issue, are other metrics collected? Do you see any error in metricbeat logs?

3

Hi @jsoriano thanks for replying.

So I've tried uncommenting that multiple times and it whenever I do I stop getting data to Kibana. When it is commented out I get data to Kibana just not with the desired results. I do however have an error in my logs:
error": {
"message": "error making http request: Post ip/jolokia/%3FignoreErrors=true&canonicalNaming=false: dial tcp (myIPhere): connect: connection refused"
}

4

This is weird because the error seems to be caused by some connectivity issue, nothing related to specific metrics. Could you paste here the configuration you have when you see this error?

5

Hi sorry I'm a bit new to this. What specific config would you like to see? My jolokia / metricbeat yml files, my code, logstash, etc. Just let me know and I'll paste it here.

6

The jolokia.yml file, but the version that produces the errors you mention.

7

Oh that version that produces the error I linked is the yml config in the initial post.

8

Oh ok, then the error seems to be caused by some connectivity issue. Can you check if your jolokia endpoint is reachable from where metricbeat is running? You can use curl for example for that.

9

{"request":{"type":"version"},"value":{"agent":"1.6.2","protocol":"7.2","config":{"listenForHttpService":"true","maxCollectionSize":"0","authIgnoreCerts":"false","agentId":"ip-jvm","debug":"false","agentType":"jvm","policyLocation":"classpath:/jolokia-access.xml","agentContext":"/jolokia","serializeException":"false","mimeType":"text/plain","maxDepth":"15","authMode":"basic","authMatch":"any","discoveryEnabled":"true","streaming":"true","canonicalNaming":"true","historyMaxEntries":"10","allowErrorDetails":"true","allowDnsReverseLookup":"true","realm":"jolokia","includeStackTrace":"true","maxObjects":"0","useRestrictorService":"false","debugMaxEntries":"100"},"info":{}},"timestamp":1563456010,"status":200}[cbarnes@centos7-6x6y8n2 modules.d]$

10

That is what I get when i curl the endpoint.

11

Here is my full json. After the initial start the error went away but I noticed my metricset.name is no longer jmx and I still have no thread count:

    {
  "_index": "metricbeat-7.1.1-2019.07.18",
  "_type": "doc",
  "_id": "bLBTBWwBdk3yjWX2WEVu",
  "_version": 1,
  "_score": null,
  "_source": {
"metricset": {
  "name": "partition"
},
"tags": [
  "beats_input_raw_event",
  "_grokparsefailure",
  "_geoip_lookup_failure"
],
"error": {
  "message": "No advertised broker with address c130sv4.uird.local:9092 found"
},
"@version": "1",
"@timestamp": "2019-07-18T13:44:02.686Z",
"ecs": {
  "version": "1.0.0"
},
"service": {
  "address": "c130sv4.uird.local:9092",
  "type": "kafka"
},
"agent": {
  "ephemeral_id": "1c93ff88-c61a-4007-82bc-12cc37fe2064",
  "version": "7.1.1",
  "id": "a3f5a0e4-17b9-4f1d-9f9f-cf7a4b3ce6bc",
  "hostname": "centos7-6x6y8n2.uird.local",
  "type": "metricbeat"
},
"event": {
  "duration": 20746091,
  "module": "kafka",
  "dataset": "kafka.partition"
},
"host": {
  "id": "7b61bfac387a44e1aea8dc8c3b6e06b0",
  "containerized": false,
  "architecture": "x86_64",
  "os": {
    "platform": "centos",
    "kernel": "3.10.0-862.11.6.el7.x86_64",
    "codename": "Core",
    "version": "7 (Core)",
    "name": "CentOS Linux",
    "family": "redhat"
  },
  "name": "centos7-6x6y8n2.uird.local",
  "hostname": "centos7-6x6y8n2.uird.local"
}
  },
  "fields": {
"@timestamp": [
  "2019-07-18T13:44:02.686Z"
]
  },
  "sort": [
1563457442686
  ]
}
12

This event is from the kafka module, not the Jolokia one.

13

Ok cool thanks for clearing that up! I really appreciate all the feedback you have been providing me.

Also once I fix that module will the thread attributes show up in the JSON? If I uncomment the code below I stop getting all data in Kibana. Commented the way it's shown will send the JSon linked above.

    - mbean: 'java.lang:type=Threading'
      attributes:
        - attr: ThreadCount
#       field: threading.thread_count
#    -mbean: 'java.lang:type=Memory'
#      attributes:
#         - attr: HeapMemoryUsage
#          field: memory.heap_usage
#        - attr: NonHeapMemoryUsage
#          field: memory.non_heap_usage
14

There seems to be some indentation problems in this file, could you comment out all the code you want and try again to see if there are more errors in the logs?

15

So if I comment everything I want out it works with no errors showing in the JSON on Kibana.

I also found out if everything works if I comment out each field. Once field is commented out I instantly get data back on Kibana. When field is uncommented I stop getting any data to Kibana.

My main goal here is to get these metrics on Kibana. I have not seen any of these metrics appear in the JSON on Kibana yet.

Hope that clears it up a bit.

16

Review the indentation of the file when you uncomment all the fields, and double check the logs for any error on startup.

17

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.