Beat to feed JMX data of Java processes to ES

Elastic Stack Beats
1

Are there already some ideas or plans to feed JMX data of Java processes into ES?

Maybe Jolokia is an option as it provides an HTTP/JSON interface to the MBeanServer within a JVM:
https://jolokia.org/reference/html/index.html

Maybe a new input_type (like http) in FileBeat where it pulls for all/filtered JMX data or a separate Beat?

Logstash itself is too heavy to install as an agent on the servers where the applications run and the jmx input plugin in logstash might not scale that good to grab the JMX data from central logstash instances from hundreds of Java processes. IMHO, the data should be fetched locally and then fed into ES (via Logstash for filtering etc.)

Thanks
Oli

2

There are no specific plans yet for JMX data. An interesting option here could be, that instead of pulling data from http, we could have an http that data can be pushed to instead of pulling. Like this the beat would not have to know all the details on how to fetch the data but it would be pushed by a tool. Not sure if something like that is supported by jolokia? The other option could be to push the data to stdin from which Filebeat is reading.

Also feel free to open a feature request.

3

Was busy with testing RC2, nightly builds recently and can now come back to this question...

I was inspired by the nginx beat which uses the same approach to pull the information. Maybe it's more an architecture decision. IMHO, well defined monitoring interfaces to pull information like http, jmx should be handled by a beat implementation itself. Then you can avoid an additional daemon running on the system which feeds the data into your proposed generic beat with an http input.

Thanks
Oli

4

@owulff Thanks for getting back and sharing your thoughts. I agree we should have as few "processes" running on the host as possible. Having a beat for this quite nice. If this is a common need it could also be that perhaps someone from the community picks it up.

5

I was looking also for a way to monitor our server farms with a beat. As there was no beat available I created the new general purpose Httpbeat. Httpbeat can call any HTTP server and send the result to the various beat outputs, e.g. logstash, elasticearch.
I'm currently using it to call Jolokia and Apache Stats.

The beat is available at https://github.com/christiangalsterer/httpbeat.
The current release is 1.0.0-beta1 and normal get/post works. There are no ready-made packages available (currently working on this) but it can be easily compiled with go.

In case of problems/questions/ideas please feel free to open an issue on github.

6

Very cool to see one more beat. We have a public list of community beats here: https://www.elastic.co/guide/en/beats/libbeat/current/community-beats.html

We would love to also add your beat to our community list. This is list is based on the following file:

Feel free to open a PR if you like to add it.

7

Httpbeat is already listed :slight_smile:

8

@christiangalsterer Seems like it was too early today in the morning :slight_smile: Sorry about that.