Configuring multiple paths in http module

[diavel (Kumaran Senapathy)]

I am trying to use http module to collect jmx metrics instead of the jolokia module because of the lack of wild card in the queries. I am having trouble configuring multiple paths.

- module: http
      metricsets: ["json"]
      enabled: true
      period: 10s
      hosts: ["localhost:8008"]
      namespace: "metrics"
      method: POST
      paths:
        - path: "/jolokia/"
          body: '{"type" : "read", "mbean" : "kafka.consumer:type=*,client-id=*", "attribute" : "bytes-consumed-rate"}'
        - path: "/jolokia/"
          body: '{"type" : "read", "mbean" : "kafka.consumer:type=*,client-id=*", "attribute" : "commit-latency-avg"}'

This does not seem to work and I see errors.

metrics.go:39: INFO Non-zero metrics in the last 30s: beat.info.uptime.ms=30000 beat.memstats.gc_next=4767600 beat.memstats.memory_alloc=4016168 beat.memstats.memory_total=47474256 libbeat.config.module.running=3 libbeat.output.read.bytes=4186 libbeat.output.write.bytes=16907 libbeat.pipeline.clients=7 libbeat.pipeline.events.active=0 libbeat.pipeline.events.published=18 libbeat.pipeline.events.total=18 libbeat.pipeline.queue.acked=18 metricbeat.http.json.events=3 metricbeat.http.json.failures=3

[ruflin]

Could you enable debug logging and share the log file here? It should give us a better picture on what happens on the http side.

[BennyInc (Benny)]

Unfortunately, multiple paths are not supported by the http module's json metricset.

What you see in the config example is for the http module's server metricset. This metricset does not query URLs. Instead it opens an http server on the specified port, and can receive input on multiple paths which are used to separate data into different namespaces.

I too want to collect JMX metrics from multiple HTTP endpoints.
As a workaround I have added multiple http module definitions with individual paths, but that results in the mess of having multiple events sent to elasticsearch - which makes it harder to correlate related items.

I believe having multiple paths in the json metricset would require a new feature to be implemented. The github issue tracker suggests to discuss the topic here, first.

[ruflin]

Thanks for bringing it up first here.

My general expectation is that the metrics belonging together are available under one endpoint / path. That is why you have to configure it multiple times. But this does not seem to be the case for you. Can you share a bit more on your data set?

You mentation that you use Http for JMX data. I wonder if this is more an enhancement request that should be handled by JMX. Work is ongoing in JMX to support wildcards.

[BennyInc (Benny)]

In our case we use IBM Liberty which provides its own JMX REST Interface - this is different from Jolokia, and does not work nicely if I grab multiple metrics at once - The HTTP Module's JSON parser cannot work with the output in those cases.

So grabbing multiple HTTP Endpoints to include in one Event would be helpful.

[ruflin]

Could you open a feature request for this? Alternatively I'm thinking if we should have a module for the above JMX rest interface instead of bending the http module to work for this.

[BennyInc (Benny)]

I've now made another attempt at grabbing multiple metrics at once from the IBM Liberty JMX REST Interface. Basically this attempt boiled down to this error message in the published event:

"error": {
  "message": "json: cannot unmarshal array into Go value of type map[string]interface {}"
}

The output from the REST request (path: "/IBMJMXConnectorREST/mbeans/WebSphere%3Atype%3DJvmStats/attributes") is this:

[{"name":"Heap","value":{"value":"1870659584","type":"java.lang.Long"}},{"name":"FreeMemory","value":{"value":"1498787264","type":"java.lang.Long"}},{"name":"GcCount","value":{"value":"631","type":"java.lang.Long"}},{"name":"UsedMemory","value":{"value":"371872320","type":"java.lang.Long"}},{"name":"ProcessCPU","value":{"value":"9.713427689998701","type":"java.lang.Double"}},{"name":"UpTime","value":{"value":"77110710","type":"java.lang.Long"}},{"name":"GcTime","value":{"value":"22053","type":"java.lang.Long"}}]

So this is a JSON Array, which cannot be unmarshalled. Further researching this, I found that you already fixed this

(Unfortunately this was a day late for the 6.2.3 release, but I'll just wait for 6.2.4 now (or do you have nightlies available somewhere for people too lazy to set up a build env? (i.e. people like me)))

[ruflin]

You might like https://beats-package-snapshots.s3.amazonaws.com/index.html Be aware that these are snapshots from master.

BTW the new feature will only be in 6.3 and not 6.2.4 as it's a bugfix release only.