Metricbeat jolokia vs Red Hat AMQ servers vs HTTP error 403

Elastic Stack Beats
1

Trying to fetch data through jolokia module from Red Hat AMQ process(es) on a host.

Have this module.d/jojokia.yml:

- module: jolokia
  metricsets: ["jmx"]
  period: 1m
  hosts: ["http://localhost:5008"]
  namespace: "metrics"
  path: "console/jolokia/?ignoreErrors=true&canonicalNaming=false"
  username: "$JOLOKIA_USER"
  password: "$JOLOKIA_PWD"
  jmx.mappings:
    - mbean: "org.apache.activemq.artemis:broker=*,component=addresses,address=*,subcomponent=queues,routing-type=*,queue=*"
      attributes:
        - attr: "MessageCount"
          field: "message.count"

only to see this in the output:

{"@timestamp":"2022-04-02T22:11:46.363Z","@metadata":{"beat":"metricbeat","type":"_doc","version":"8.1.0"},"event":{"dataset":"jolokia.jmx","module":"jolokia","duration":2146134},"metricset":{"period":60000,"name":"jmx"},"service":{"address":"http://localhost:5008/console/jolokia/%3FignoreErrors=true&canonicalNaming=false","type":"jolokia"},"error":{"message":"HTTP error 403 in : 403 Forbidden"},"ecs":{"version":"8.0.0"},"host":{"name":"test_metricbeat"},"agent":{"name":"test_metricbeat","type":"metricbeat","version":"8.1.0","ephemeral_id":"048ee1f0-41b0-4943-9845-d46d848fac3c","id":"e4e7923d-ae95-45b8-aeb0-cda25d728d70"}}

and this is the debug log output:

{"log.level":"debug","@timestamp":"2022-04-03T00:11:46.362+0200","log.logger":"reload","log.origin":{"file.name":"cfgfile/list.go","file.line":81},"message":"Start list: 1, Stop list: 0","service.name":"me
tricbeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-04-03T00:11:46.363+0200","log.logger":"reload","log.origin":{"file.name":"cfgfile/list.go","file.line":105},"message":"Starting runner: RunnerGroup{jolokia [metricse
ts=1]}","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-04-03T00:11:46.363+0200","log.logger":"module","log.origin":{"file.name":"module/wrapper.go","file.line":127},"message":"Starting Wrapper[name=jolokia, len(metricSet
Wrappers)=1]","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-04-03T00:11:46.363+0200","log.origin":{"file.name":"cfgfile/reload.go","file.line":224},"message":"Loading of config files completed.","service.name":"metricbeat","ec
s.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-04-03T00:11:46.363+0200","log.logger":"module","log.origin":{"file.name":"module/wrapper.go","file.line":189},"message":"Starting metricSetWrapper[module=jolokia, na
me=jmx, host=localhost:5008]","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-04-03T00:11:46.363+0200","log.logger":"jolokia.jmx","log.origin":{"file.name":"jmx/config.go","file.line":484},"message":"Jolokia request URI and body","service.name
":"metricbeat","host":"localhost:5008","httpMethod":"POST","URI":"","body":"[{\"type\":\"read\",\"mbean\":\"org.apache.activemq.artemis:address=*,broker=*,component=addresses,queue=*,routing-type=*,subcomp
onent=queues\",\"attribute\":[\"MessageCount\"],\"config\":{\"canonicalNaming\":true,\"ignoreErrors\":true}}]","type":"request","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2022-04-03T00:11:46.365+0200","log.origin":{"file.name":"module/wrapper.go","file.line":254},"message":"Error fetching data for metricset jolokia.jmx: HTTP error 403 in :
 403 Forbidden","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-04-03T00:11:46.365+0200","log.logger":"processors","log.origin":{"file.name":"processing/processors.go","file.line":209},"message":"Publish event: {\n  \"@timestamp\
": \"2022-04-02T22:11:46.363Z\",\n  \"@metadata\": {\n    \"beat\": \"metricbeat\",\n    \"type\": \"_doc\",\n    \"version\": \"8.1.0\"\n  },\n  \"agent\": {\n    \"name\": \"test_metricbeat\",\n    \"typ
e\": \"metricbeat\",\n    \"version\": \"8.1.0\",\n    \"ephemeral_id\": \"048ee1f0-41b0-4943-9845-d46d848fac3c\",\n    \"id\": \"e4e7923d-ae95-45b8-aeb0-cda25d728d70\"\n  },\n  \"event\": {\n    \"dataset
\": \"jolokia.jmx\",\n    \"module\": \"jolokia\",\n    \"duration\": 2146134\n  },\n  \"metricset\": {\n    \"name\": \"jmx\",\n    \"period\": 60000\n  },\n  \"service\": {\n    \"address\": \"http://loc
alhost:5008/console/jolokia/%3FignoreErrors=true&canonicalNaming=false\",\n    \"type\": \"jolokia\"\n  },\n  \"error\": {\n    \"message\": \"HTTP error 403 in : 403 Forbidden\"\n  },\n  \"ecs\": {\n    \
"version\": \"8.0.0\"\n  },\n  \"host\": {\n    \"name\": \"test_metricbeat\"\n  }\n}","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-04-03T00:11:47.366+0200","log.logger":"publisher","log.origin":{"file.name":"memqueue/ackloop.go","file.line":160},"message":"ackloop: receive ack [0: 0, 1]","servic
e.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-04-03T00:11:47.366+0200","log.logger":"publisher","log.origin":{"file.name":"memqueue/ackloop.go","file.line":128},"message":"ackloop: return ack to broker loop:1","
service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-04-03T00:11:47.366+0200","log.logger":"publisher","log.origin":{"file.name":"memqueue/ackloop.go","file.line":131},"message":"ackloop:  done send ack","service.name"
:"metricbeat","ecs.version":"1.6.0"}

this is working from curl:

curl -o /tmp/curl.output 'http://$JOLOKIA_USER:$JOLOKIA_PWD@localhost:5008/console/jolokia/read/org.apache.activemq.artemis:broker="*",component=addresses,address="*",subcomponent=queues,routing-type="*",queue="*"/MessageCount'
2

Perhaps You are missing the leading /on the path?

path: "/console/jolokia/?ignoreErrors=true&canonicalNaming=false"

3

Thanks, but had that initially, makes no difference, also seen from the log:

also tried without the options ignoreErrors and canonicalNaming, also no difference, also tried with console/jolokia/read in path, also no difference

4

Did you test hard coding the username and password in the metricbest configuration?

Ohh and I think the syntax per doc is

username: ${JOLOKIA_USER}

5

@stephenb :+1:t2: U'R bang on, thanks! Missed the curly brackets to properly fetch from my keystore. Would have thought to have gotten a http error 401 though instead of 403 :roll_eyes:

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.