Error creating runner from config: 1 error: no metricsets configured for module 'jolokia'

Elastic Stack Beats
1

Unable to send Jolokia metrics, I get this error in my /var/log/metricbeat/metricbeat error log

2019-02-14T14:52:26.415Z	ERROR	[reload]	cfgfile/list.go:104	Error creating runner from config: 1 error: no metricsets configured for module 'jolokia'

I've tried a lot of different configurations, and I know I can reach the server. It is sending some information but not the jolokia metrics. I guess metricbeat sends some other information but all I care about is Jolokia. Does anyone know what this error means? I saw this question has been asked before but it was closed and it had no answer.

metricbeat.config.modules:
  # Glob pattern for configuration loading
  path: ${path.config}/modules.d/*.yml

  # Set to true to enable config reloading
  reload.enabled: true

  # Period on which files under path should be checked for changes
  #reload.period: 10s
metricbeat.modules:
  - module: "jolokia"
    metricsets: ["jmx"]
    enabled: true
    period: 10s
    namespace: "jolokia_metrics"
    hosts: ["https://xxx.xxx.xxx.xxx:8443"]
    path: "/jolokia"
    username: "tomcat"
    password: "tomcat"
    jmx.mappings:
    - mbean: 'java.lang:type=Runtime'
    attributes:
    - attr: Uptime
      field: uptime
    - mbean: 'java.lang:type=GarbageCollector,name=ConcurrentMarkSweep'
    attributes:
    - attr: CollectionTime
      field: gc.cms_collection_time
    - attr: CollectionCount
      field: gc.cms_collection_count
    - mbean: 'java.lang:type=Memory'
    attributes:
    - attr: HeapMemoryUsage
      field: memory.heap_usage
    - attr: NonHeapMemoryUsage
      field: memory.non_heap_usage
    - mbean: 'java.lang:type=Threading'
    attributes:
    - attr: ThreadCount
      field: threadcount.threadcount
    - attr: DaemonThreadCount
      field: threadcount.daemon

setup.template.settings:
  index.number_of_shards: 1
  index.codec: best_compression
  #_source.enabled: false
setup.kibana:
output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["xxx.xxx.xxx.xxx:9200"]
2

The above means that metricsets: ["jmx"] is empty. It looks ok here so I wonder what you have in your modules.d/ direcotry?

It could also be an indentation issue in the YAML file. Could you try remove two spaces in front of all the config lines under metricbeat.modules: ?

3

Not sure what I'm supposed to see in that directory but it does contain a jolokia yml file. I tried the changing of the spaces in the file like you said but it's the same issue here is what the file looks like now:

metricbeat.modules:
- module: "jolokia"
  metricsets: ["jmx"]
  enabled: true
  period: 10s
  namespace: "jolokia_metrics"
  hosts: ["xxx.xxx.xxx.xxx:8443"]
  username: "tomcat"
  password: "tomcat"
  path: "/jolokia"
  jmx.mappings:
  - mbean: 'java.lang:type=Runtime'
  attributes:
  - attr: Uptime
  field: uptime
  - mbean: 'java.lang:type=GarbageCollector,name=ConcurrentMarkSweep'
  attributes:
  - attr: CollectionTime
  field: gc.cms_collection_time
  - attr: CollectionCount
  field: gc.cms_collection_count
  - mbean: 'java.lang:type=Memory'
  attributes:
  - attr: HeapMemoryUsage
  field: memory.heap_usage
  - attr: NonHeapMemoryUsage
  field: memory.non_heap_usage
  - mbean: 'java.lang:type=Threading'
  attributes:
  - attr: ThreadCount
  field: threadcount.threadcount
  - attr: DaemonThreadCount
  field: threadcount.daemon

Note: It would not work unless the field tags were indented two spaces more than the -attr tag.
Side Question: Is there a way to configure this using something that is NOT space sensitive because if this is really the problem, I'd much rather never use YAML.

Also, I'm not sure I understand what namespace or a metricset are in the context of Jolokia.

If I curl the jolokia endpoint like this I get output that I am expecting via JSON:

[root@josebox metricbeat]# curl -u tomcat:tomcat https://xxx.xxx.xxx.xxx:8443/jolokia -lk

{
  "request": {
    "type": "version"
  },
  "value": {
    "agent": "1.6.0",
    "protocol": "7.2",
    "config": {
      "listenForHttpService": "true",
      "maxCollectionSize": "0",
      "authIgnoreCerts": "false",
      "agentId": "192.168.250.90-30871-38f51807-servlet",
      "agentType": "servlet",
      "policyLocation": "classpath:/jolokia-access.xml",
      "agentContext": "/jolokia",
      "mimeType": "text/plain",
      "discoveryEnabled": "false",
      "streaming": "true",
      "historyMaxEntries": "10",
      "allowDnsReverseLookup": "true",
      "maxObjects": "0",
      "debug": "false",
      "serializeException": "false",
      "detectorOptions": "{}",
      "dispatcherClasses": "org.jolokia.http.Jsr160ProxyNotEnabledByDefaultAnymoreDispatcher",
      "maxDepth": "15",
      "authMode": "basic",
      "canonicalNaming": "true",
      "allowErrorDetails": "true",
      "realm": "jolokia",
      "includeStackTrace": "true",
      "useRestrictorService": "false",
      "debugMaxEntries": "100"
    },
    "info": {
      "product": "tomcat",
      "vendor": "Apache",
      "version": "9.0.6"
    }
  },
  "timestamp": 1550163092,
  "status": 200
}

From the logs, I can see that metricbeat is working but not for jolokia:

2019-02-14T17:06:27.975Z	INFO	[monitoring]	log/log.go:141	Non-zero metrics in the last 30s	{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":1270,"time":{"ms":85}},"total":{"ticks":3090,"time":{"ms":195},"value":3090},"user":{"ticks":1820,"time":{"ms":110}}},"info":{"ephemeral_id":"91f20b5d-e1f4-427b-b199-4ac840a2441f","uptime":{"ms":480033}},"memstats":{"gc_next":6396128,"memory_alloc":3494632,"memory_total":419299408,"rss":24576}},"libbeat":{"config":{"module":{"running":0},"reloads":3},"output":{"events":{"acked":42,"batches":6,"total":42},"read":{"bytes":2287},"write":{"bytes":32593}},"pipeline":{"clients":4,"events":{"active":0,"published":42,"total":42},"queue":{"acked":42}}},"metricbeat":{"jolokia":{"jmx":{"events":3,"failures":3}},"system":{"cpu":{"events":3,"success":3},"load":{"events":3,"success":3},"memory":{"events":3,"success":3},"network":{"events":6,"success":6},"process":{"events":21,"success":21},"process_summary":{"events":3,"success":3}}},"system":{"load":{"1":0.08,"15":0.05,"5":0.03,"norm":{"1":0.04,"15":0.025,"5":0.015}}}}}}
2019-02-14T17:06:28.061Z	ERROR	[reload]	cfgfile/list.go:104	Error creating runner from config: 1 error: no metricsets configured for module 'jolokia'
2019-02-14T17:06:38.063Z	ERROR	[reload]	cfgfile/list.go:104	Error creating runner from config: 1 error: no metricsets configured for module 'jolokia'
2019-02-14T17:06:48.064Z	ERROR	[reload]	cfgfile/list.go:104	Error creating runner from config: 1 error: no metricsets configured for module 'jolokia'
4

Also tried it with the following config, valid enough to make it start up but still gives the same error:

metricbeat.modules:
- module: "jolokia"
  metricsets: ["jmx"]
  enabled: true
  period: 10s
  namespace: "jolokia_metrics"
  hosts: ["xxx.xxx.xxx.xxx:8443"]
  username: "tomcat"
  password: "tomcat"
  path: "/jolokia"
  jmx.mappings:
    - mbean: 'java.lang:type=Runtime'
      attributes:
      - attr: Uptime
        field: uptime
    - mbean: 'java.lang:type=GarbageCollector,name=ConcurrentMarkSweep'
      attributes:
      - attr: CollectionTime
        field: gc.cms_collection_time
      - attr: CollectionCount
        field: gc.cms_collection_count
    - mbean: 'java.lang:type=Memory'
      attributes:
      - attr: HeapMemoryUsage
        field: memory.heap_usage
      - attr: NonHeapMemoryUsage
        field: memory.non_heap_usage
    - mbean: 'java.lang:type=Threading'
      attributes:
      - attr: ThreadCount
        field: threadcount.threadcount
      - attr: DaemonThreadCount
        field: threadcount.daemon
5

Do you have a verified working config that you guys use? I've tried several configs/things I've seen on the forums/documentation. If you have one that is known to work with a default installation of Tomcat/Jolokia WAR file that would be super helpful. I imagine you guys would have one for testing?

6

Don't know if this helps anyone debug but the file /etc/metricbeat/modules.d/jolokia.yml looks like the following:

[root@jxt163-grm-rundeckpro-env metricbeat]# cat /etc/metricbeat/modules.d/jolokia.yml
# Module: jolokia
# Docs: https://www.elastic.co/guide/en/beats/metricbeat/6.4/metricbeat-module-jolokia.html

- module: jolokia
  #metricsets: ["jmx"]
  period: 10s
  hosts: ["localhost"]
  namespace: "metrics"
  #path: "/jolokia/?ignoreErrors=true&canonicalNaming=false"
  #username: "user"
  #password: "secret"
  jmx.mappings:
    #- mbean: 'java.lang:type=Runtime'
    #  attributes:
    #    - attr: Uptime
    #      field: uptime
    #- mbean: 'java.lang:type=Memory'
    #  attributes:
    #    - attr: HeapMemoryUsage
    #      field: memory.heap_usage
    #    - attr: NonHeapMemoryUsage
    #      field: memory.non_heap_usage
    # GC Metrics - this depends on what is available on your JVM
    #- mbean: 'java.lang:type=GarbageCollector,name=ConcurrentMarkSweep'
    #  attributes:
    #    - attr: CollectionTime
    #      field: gc.cms_collection_time
    #    - attr: CollectionCount
    #      field: gc.cms_collection_count

  jmx.application:
  jmx.instance:
7

Removing the entry in the previous file /etc/metricbeat/modules.d/jolokia.yml and replacing with my config seems to have made it start shipping my jolokia data. However, now it is giving a 404 error.

/etc/metricbeat/modules.d/jolokia.yml

- module: "jolokia"
  metricsets: ["jmx"]
  enabled: true
  period: 10s
  namespace: "jolokia_metrics"
  hosts: ["xxx.xxx.xxx.xxx:8443"]
  username: "tomcat"
  password: "tomcat"
  path: "/jolokia/?ignoreErrors=true&canonicalNaming=false"
  ssl.verification_mode: "none"
  jmx.mappings:
    - mbean: 'java.lang:type=Runtime'
      attributes:
        - attr: Uptime
          field: uptime
    - mbean: 'java.lang:type=GarbageCollector,name=ConcurrentMarkSweep'
      attributes:
        - attr: CollectionTime
          field: gc.cms_collection_time
        - attr: CollectionCount
          field: gc.cms_collection_count
    - mbean: 'java.lang:type=Memory'
      attributes:
        - attr: HeapMemoryUsage
          field: memory.heap_usage
        - attr: NonHeapMemoryUsage
          field: memory.non_heap_usage
    - mbean: 'java.lang:type=Threading'
      attributes:
        - attr: ThreadCount
          field: threadcount.threadcount
        - attr: DaemonThreadCount
          field: threadcount.daemon

However, now on the kibana server I'm seeing the following:

{
  "_index": "metricbeat-6.4.2-2019.02.14",
  "_type": "doc",
  "_id": "vSqU7WgBx3qTvatO0eeH",
  "_version": 1,
  "_score": null,
  "_source": {
    "@timestamp": "2019-02-14T19:56:18.716Z",
    "beat": {
      "name": "josebox.com",
      "hostname": "josebox.com",
      "version": "6.4.2"
    },
    "host": {
      "name": "josebox.com"
    },
    "metricset": {
      "module": "jolokia",
      "host": "xxx.xxx.xxx.xxx:8443",
      "rtt": 1116,
      "name": "jmx"
    },
    "error": {
      "message": "HTTP error 400 in jmx: 400 "
    }
  },
  "fields": {
    "@timestamp": [
      "2019-02-14T19:56:18.716Z"
    ]
  },
  "highlight": {
    "metricset.module": [
      "@kibana-highlighted-field@jolokia@/kibana-highlighted-field@"
    ]
  },
  "sort": [
    1550174178716
  ]
}

This is why I don't understand what a namespace is, is it arbitrary for elastic's usage only or is metricbeat trying to hit some kind of endpoint? It's not clear in the documentation and some of the documentation has #metricsets ['jmx'] ( a comment, which means it doesn't matter?)

8

Found the solution, for whatever reason remove everything in the /etc/metricbeat/metricbeat.yml that has nothing to do with my configuration. Also Jolokia isn't smart enough to follow redirects so if you're using https, you HAVE to specify https in the hosts field.

metricbeat.modules:
  - module: jolokia
    metricsets: ["jmx"]
    enabled: true
    period: 10s
    hosts: ["https://localhost:8443"]
    path: "/jolokia/?ignoreErrors=true&canonicalNaming=false"
    username: "tomcat"
    password: "tomcat"
    ssl.verification_mode: "none"
    namespace: "metrics"
    jmx.mappings:
      - mbean: 'java.lang:type=Runtime'
        attributes:
          - attr: Uptime
            field: uptime
      - mbean: 'java.lang:type=Threading'
        attributes:
          - attr: ThreadCount
            field: threadcount.threadcount
          - attr: DaemonThreadCount
            field: threadcount.daemon
      - mbean: 'java.lang:type=GarbageCollector,name=ConcurrentMarkSweep'
        attributes:
          - attr: CollectionTime
            field: gc.cms_collection_time
          - attr: CollectionCount
            field: gc.cms_collection_count
      - mbean: 'java.lang:type=Memory'
        attributes:
          - attr: HeapMemoryUsage
            field: memory.heap_usage
          - attr: NonHeapMemoryUsage
            field: memory.non_heap_usage
    jmx.application:
    jmx.instance:
output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["xxx.xxx.xxx.xxx:9200"]
9

:+1:

The metricsets line is commented out by default. So if you want to collect jmx metrics, after enabling the jolokia module, you also have to go to the modules.d/jolokia.yml file and uncomment this line. Sometimes, for example in system module, there are more than one metricset. Some of them are by default enabled and some are not. The ones are not enabled by default are the ones got commented out.

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.