Metricbeat jolokia module: "Cannot unmarshal json response"

[gregbk (greg)]

Hello Beats team!

I am trying the Jolokia module on 2 different jmx env.

I got for both the same error message in kibana:
Cannot unmarshal json response: invalid character '\x00' looking for beginning of value

I use that docker to simulate a jmx:

FROM java:8
MAINTAINER "Alexei Novikov <alexeinov@gmail.com>"
COPY . /usr/src/myapp
WORKDIR /usr/src/myapp
RUN javac Hammer.java
CMD ["java", \
"-Dcom.sun.management.jmxremote", \
"-Dcom.sun.management.jmxremote.port=9010", \
# Uncomment this line when running JMX console on a different host
#"-Dcom.sun.management.jmxremote.rmi.port=9010", \
"-Dcom.sun.management.jmxremote.local.only=false", \
"-Dcom.sun.management.jmxremote.authenticate=false", \
"-Dcom.sun.management.jmxremote.ssl=false", \
# Uncomment this line when running JMX console on a different host, use a name of the host where Docker is running
#"-Djava.rmi.server.hostname=192.168.0.101", \
"Hammer"]
EXPOSE 9010

And this minimum metricbeat conf:

metricbeat.modules:

- module: jolokia
  metricsets: ["jmx"]
  enabled: true
  period: 1s
  hosts: ["jmx:9010"]
  namespace: "metrics"
  #path: "/jolokia/?ignoreErrors=true&canonicalNaming=true"
  jmx.mappings:
    - mbean: 'java.lang:type=Runtime'
      attributes:
        - attr: Uptime
          field: uptime

Event is send and coming to the right index
fetches.jolokia-jmx.events=30 fetches.jolokia-jmx.failures=30

but with error: Cannot unmarshal json response: invalid character '\x00' looking for beginning of value

I am on 5.4.1 for elk+metricbeat.
On a totally different jmx server in another env, I got the same error. Probably my metricbeat/(es template?) is missing something...
I tried to send to logstash and directly to es: same result.

Similar to Metricbeat 5.4.0 Jolokia module

Thank you for your help & good weekend!
Greg.

[andrewkroh]

If I understand your setup correctly, you are trying to communicate directly to the JMX endpoint in the JVM. Metricbeat needs to communicate to the Jolokia REST endpoint. Jolokia acts as a bridge between Metricbeat and JMX and exposes a HTTP/REST/JSON interface to JMX.

[joshuaclark (Josh)]

You need to follow the steps from this document https://jolokia.org/reference/html/agents.html#agents-jvm 3.4.1.1. Installation section. then set metricbeat to connect to the JVM host and the port you set for the Jolokia agent. You then would be able to connect to http://host:7777/jolokia/read/java.lang:name=PS%20Scavenge,type=GarbageCollector etc.

[gregbk (greg)]

Thanks a lot both!
Yes, I had in mind that I needed to connect to jmx only while jolokia agent only can be understood by metricbeat...

Here is the right setup:

FROM java:8
MAINTAINER "Alexei Novikov <alexeinov@gmail.com>"
COPY . /usr/src/myapp
WORKDIR /usr/src/myapp
RUN javac Hammer.java

ENV JOLOKIA_VERSION 1.3.1
RUN mkdir /opt/jolokia && wget http://central.maven.org/maven2/org/jolokia/jolokia-jvm/1.3.1/jolokia-jvm-1.3.1-agent.jar -O /opt/jolokia/jolokia.jar

# Print out the version
RUN java -jar /opt/jolokia/jolokia.jar --version

CMD ["java", \
"-javaagent:/opt/jolokia/jolokia.jar=port=8778,host=0.0.0.0", \
"-Dcom.sun.management.jmxremote", \
"-Dcom.sun.management.jmxremote.port=9010", \
# Uncomment this line when running JMX console on a different host
#"-Dcom.sun.management.jmxremote.rmi.port=9010", \
"-Dcom.sun.management.jmxremote.local.only=false", \
"-Dcom.sun.management.jmxremote.authenticate=false", \
"-Dcom.sun.management.jmxremote.ssl=false", \
# Uncomment this line when running JMX console on a different host, use a name of the host where Docker is running
#"-Djava.rmi.server.hostname=192.168.0.101", \
"Hammer"]
EXPOSE 9010 8778

- module: jolokia
  metricsets: ["jmx"]
  enabled: true
  period: 1s
  hosts: ["jmx:8778"]
  namespace: "metrics"
  path: "/jolokia/?ignoreErrors=true&canonicalNaming=true"
  jmx.mappings:
    - mbean: 'java.lang:type=ClassLoading'
      attributes:
        - attr: LoadedClassCount
          field: loaded_class_count
    - mbean: 'java.lang:name=PS Scavenge,type=GarbageCollector'
      attributes:
        - attr: CollectionTime
          field: gc.collection_time
        - attr: CollectionCount
          field: gc.collection_count
    - mbean: 'java.lang:type=Memory'
      attributes:
        - attr: HeapMemoryUsage
          field: heap_memory_usage
        - attr: NonHeapMemoryUsage
          field: non_heap_memory_usage
    - mbean: 'java.lang:type=OperatingSystem'
      attributes:
        - attr: ProcessCpuLoad
          field: process_cpu_load
    - mbean: 'java.lang:type=Runtime'
      attributes:
        - attr: Uptime
          field: uptime
    - mbean: 'java.lang:type=Threading'
      attributes:
        - attr: ThreadCount
          field: thread_count

Thx!