404 from jolokia module


(ethr bunny) #1

I had a working setup with cassandra + jmx + jolokia + metricbeat not too long ago. Lately Im only getting the message : "HTTP Error 404 in jmx: 404 not found". Presently runnign 6.4. Have tried downgrading to 6.3 to no avail.

What Ive tested:

  • I can get jmx metrics using the jmx cli
  • I can get jolokia values using curl

All that leaves is the metric beat module.

Suggestions on where Ive gone wrong?

---- EDIT:

jolokia.yml:

  • module: jolokia
    metricsets: ["jmx"]
    hosts: ["localhost:7770"]
    namespace: "jmxmetrics"
    jmx.mappings:
    • mbean: 'org.apache.cassandra.metrics:type=Compaction,name=PendingTasks'
      attributes:
      • attr: Value
        field: compaction.pending_tasks

metricbeat.yml:
metricbeat.config.modules:
path: ${path.config}/modules.d/*.yml

reload.enabled: false

setup.template.settings:
index.number_of_shards: 1
index.codec: best_compression

output.logstash:
hosts: ["logstash.myhost:5044"]

Log output from metricbeat:
.

2018-09-07T13:44:14.747Z INFO [monitoring] log/log.go:149 Total non-zero metrics {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":1110,"time":{"ms":1110}},"total":{"ticks":3140,"time":{"ms":3144},"value":3140},"user":{"ticks":2030,"time":{"ms":2034}}},"info":{"ephemeral_id":"b9c4295b-e39a-4da8-8762-90a4c126a27d","uptime":{"ms":68100}},"memstats":{"gc_next":8702976,"memory_alloc":4399120,"memory_total":140195344,"rss":33996800}},"libbeat":{"config":{"module":{"running":0},"reloads":1},"output":{"events":{"acked":141,"batches":7,"total":141},"read":{"bytes":54},"type":"logstash","write":{"bytes":31131}},"pipeline":{"clients":4,"events":{"active":0,"published":141,"retry":29,"total":141},"queue":{"acked":141}}},"metricbeat":{"jolokia":{"jmx":{"events":7,"failures":7}},"system":{"cpu":{"events":7,"success":7},"filesystem":{"events":16,"success":16},"fsstat":{"events":2,"success":2},"load":{"events":7,"success":7},"memory":{"events":7,"success":7},"network":{"events":28,"success":28},"process":{"events":59,"success":59},"process_summary":{"events":7,"success":7},"uptime":{"events":1,"success":1}}},"system":{"cpu":{"cores":16},"load":{"1":0.17,"15":0.14,"5":0.16,"norm":{"1":0.0106,"15":0.0088,"5":0.01}}}}}}

curl:
$ curl http://localhost:7770/jolokia/read/org.apache.cassandra.metrics:type=ClientRequest,scope=Write,name=Latency

{"request":{"mbean":"org.apache.cassandra.metrics:name=Latency,scope=Write,type=ClientRequest","type":"read"},"value":{"Mean":916.8661242633178,"75thPercentile":943.1270000000001,"StdDev":156.89921270826937,"98thPercentile":1358.102,"RateUnit":"events/second","95thPercentile":1131.752,"99thPercentile":1358.102,"Max":2346.799,"Count":4489,"FiveMinuteRate":0.9743310708230863,"50thPercentile":943.1270000000001,"MeanRate":0.6436780814230763,"Min":454.827,"OneMinuteRate":0.9598666879425546,"DurationUnit":"microseconds","999thPercentile":1358.102,"FifteenMinuteRate":0.8846616462455071},"timestamp":1536328078,"status":200}


(ethr bunny) #2

After a dive into the metricbeat code and subsequent experiments with curl POST requests Im thinking this might be a jolokia issue after all. jolokia works fine with GET requests but not POST.

Im switching to an HTTP module instead. Seems to be working ok --> and --> it takes wildcards!


(Mario Castro) #3

Hi @ethrbunny :slight_smile:

Jolokia modules uses POST as you can see in the following line: https://github.com/elastic/beats/blob/master/metricbeat/module/jolokia/jmx/jmx.go#L83

Did you have problems by doing curl and POST on Jolokia? Please, can you develop a bit? If it's an issue with the module we'll try to fix it to help the community :slight_smile: Even better, we encourage you to fix it if you feel that you understand the root of the problem :slight_smile:


(ethr bunny) #4

I was unable to get POST to work with jolokia. Are there some specific examples of curl usage so I can make sure Im doing it correctly?

In any event the 'http/json' module is a good replacement.


(Guido Navalesi) #5

Hi! I have the same problem. This configuration was working with v6.2.4 until I upgraded to v6.4.0:

- module: jolokia
  metricsets: ["jmx"]
  enabled: true
  period: 30s
  namespace: "java_jmx"
  hosts: ["localhost:8778"]
  jmx.mappings:
  - mbean: 'org.apache.cassandra.metrics:name=HitRate,scope=KeyCache,type=Cache'
    attributes:
    - attr: Value
      field: key_cache.hit_rate

I've been able to make a POST with curl with the body that I've found in the logs:
curl --header "Content-Type: application/json" --request POST --data '{"type":"read","mbean":"org.apache.cassandra.metrics:name=HitRate,scope=KeyCache,type=Cache","attribute":["Value"],"config":{"canonicalNaming":true,"ignoreErrors":true}}' http://localhost:8778/jolokia/

{"request":{"mbean":"org.apache.cassandra.metrics:name=HitRate,scope=KeyCache,type=Cache","attribute":"Value","type":"read"},"value":{"Value":0.9983935115946554},"timestamp":1538180662,"status":200}


(Guido Navalesi) #6

Fixed! I had to add the /jolokia/ path to the host:

- module: jolokia
  metricsets: ["jmx"]
  enabled: true
  period: 30s
  namespace: "java_jmx"
  hosts: ["localhost:8778/jolokia/"]
  jmx.mappings:
  - mbean: 'org.apache.cassandra.metrics:name=HitRate,scope=KeyCache,type=Cache'
    attributes:
    - attr: Value
      field: key_cache.hit_rate

(ruflin) #7

Thanks for posting the solution here.


(system) #8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.