404 from jolokia module

[ethrbunny]

I had a working setup with cassandra + jmx + jolokia + metricbeat not too long ago. Lately Im only getting the message : "HTTP Error 404 in jmx: 404 not found". Presently runnign 6.4. Have tried downgrading to 6.3 to no avail.

What Ive tested:

• I can get jmx metrics using the jmx cli
• I can get jolokia values using curl

All that leaves is the metric beat module.

Suggestions on where Ive gone wrong?

---- EDIT:

jolokia.yml:

• module: jolokia
  metricsets: ["jmx"]
  hosts: ["localhost:7770"]
  namespace: "jmxmetrics"
  jmx.mappings:
  • mbean: 'org.apache.cassandra.metrics:type=Compaction,name=PendingTasks'
    attributes:
    • attr: Value
      field: compaction.pending_tasks

metricbeat.yml:
metricbeat.config.modules:
path: ${path.config}/modules.d/*.yml

reload.enabled: false

setup.template.settings:
index.number_of_shards: 1
index.codec: best_compression

output.logstash:
hosts: ["logstash.myhost:5044"]

Log output from metricbeat:
.

2018-09-07T13:44:14.747Z INFO [monitoring] log/log.go:149 Total non-zero metrics {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":1110,"time":{"ms":1110}},"total":{"ticks":3140,"time":{"ms":3144},"value":3140},"user":{"ticks":2030,"time":{"ms":2034}}},"info":{"ephemeral_id":"b9c4295b-e39a-4da8-8762-90a4c126a27d","uptime":{"ms":68100}},"memstats":{"gc_next":8702976,"memory_alloc":4399120,"memory_total":140195344,"rss":33996800}},"libbeat":{"config":{"module":{"running":0},"reloads":1},"output":{"events":{"acked":141,"batches":7,"total":141},"read":{"bytes":54},"type":"logstash","write":{"bytes":31131}},"pipeline":{"clients":4,"events":{"active":0,"published":141,"retry":29,"total":141},"queue":{"acked":141}}},"metricbeat":{"jolokia":{"jmx":{"events":7,"failures":7}},"system":{"cpu":{"events":7,"success":7},"filesystem":{"events":16,"success":16},"fsstat":{"events":2,"success":2},"load":{"events":7,"success":7},"memory":{"events":7,"success":7},"network":{"events":28,"success":28},"process":{"events":59,"success":59},"process_summary":{"events":7,"success":7},"uptime":{"events":1,"success":1}}},"system":{"cpu":{"cores":16},"load":{"1":0.17,"15":0.14,"5":0.16,"norm":{"1":0.0106,"15":0.0088,"5":0.01}}}}}}

curl:
$ curl http://localhost:7770/jolokia/read/org.apache.cassandra.metrics:type=ClientRequest,scope=Write,name=Latency

{"request":{"mbean":"org.apache.cassandra.metrics:name=Latency,scope=Write,type=ClientRequest","type":"read"},"value":{"Mean":916.8661242633178,"75thPercentile":943.1270000000001,"StdDev":156.89921270826937,"98thPercentile":1358.102,"RateUnit":"events/second","95thPercentile":1131.752,"99thPercentile":1358.102,"Max":2346.799,"Count":4489,"FiveMinuteRate":0.9743310708230863,"50thPercentile":943.1270000000001,"MeanRate":0.6436780814230763,"Min":454.827,"OneMinuteRate":0.9598666879425546,"DurationUnit":"microseconds","999thPercentile":1358.102,"FifteenMinuteRate":0.8846616462455071},"timestamp":1536328078,"status":200}

[ethrbunny]

After a dive into the metricbeat code and subsequent experiments with curl POST requests Im thinking this might be a jolokia issue after all. jolokia works fine with GET requests but not POST.

Im switching to an HTTP module instead. Seems to be working ok --> and --> it takes wildcards!

[Mario_Castro]

Hi @ethrbunny

Jolokia modules uses POST as you can see in the following line: https://github.com/elastic/beats/blob/master/metricbeat/module/jolokia/jmx/jmx.go#L83

Did you have problems by doing curl and POST on Jolokia? Please, can you develop a bit? If it's an issue with the module we'll try to fix it to help the community Even better, we encourage you to fix it if you feel that you understand the root of the problem

[ethrbunny]

I was unable to get POST to work with jolokia. Are there some specific examples of curl usage so I can make sure Im doing it correctly?

In any event the 'http/json' module is a good replacement.

[Guido_Navalesi]

Hi! I have the same problem. This configuration was working with v6.2.4 until I upgraded to v6.4.0:

- module: jolokia
  metricsets: ["jmx"]
  enabled: true
  period: 30s
  namespace: "java_jmx"
  hosts: ["localhost:8778"]
  jmx.mappings:
  - mbean: 'org.apache.cassandra.metrics:name=HitRate,scope=KeyCache,type=Cache'
    attributes:
    - attr: Value
      field: key_cache.hit_rate

I've been able to make a POST with curl with the body that I've found in the logs:
curl --header "Content-Type: application/json" --request POST --data '{"type":"read","mbean":"org.apache.cassandra.metrics:name=HitRate,scope=KeyCache,type=Cache","attribute":["Value"],"config":{"canonicalNaming":true,"ignoreErrors":true}}' http://localhost:8778/jolokia/

{"request":{"mbean":"org.apache.cassandra.metrics:name=HitRate,scope=KeyCache,type=Cache","attribute":"Value","type":"read"},"value":{"Value":0.9983935115946554},"timestamp":1538180662,"status":200}

[Guido_Navalesi]

Fixed! I had to add the /jolokia/ path to the host:

- module: jolokia
  metricsets: ["jmx"]
  enabled: true
  period: 30s
  namespace: "java_jmx"
  hosts: ["localhost:8778/jolokia/"]
  jmx.mappings:
  - mbean: 'org.apache.cassandra.metrics:name=HitRate,scope=KeyCache,type=Cache'
    attributes:
    - attr: Value
      field: key_cache.hit_rate

[ruflin]

Thanks for posting the solution here.