Beat to feed JMX data of Java processes to ES

owulff · November 12, 2015, 9:50pm

Are there already some ideas or plans to feed JMX data of Java processes into ES?

Maybe Jolokia is an option as it provides an HTTP/JSON interface to the MBeanServer within a JVM:
https://jolokia.org/reference/html/index.html

Maybe a new input_type (like http) in FileBeat where it pulls for all/filtered JMX data or a separate Beat?

Logstash itself is too heavy to install as an agent on the servers where the applications run and the jmx input plugin in logstash might not scale that good to grab the JMX data from central logstash instances from hundreds of Java processes. IMHO, the data should be fetched locally and then fed into ES (via Logstash for filtering etc.)

Thanks
Oli

ruflin · November 13, 2015, 7:28am

There are no specific plans yet for JMX data. An interesting option here could be, that instead of pulling data from http, we could have an http that data can be pushed to instead of pulling. Like this the beat would not have to know all the details on how to fetch the data but it would be pushed by a tool. Not sure if something like that is supported by jolokia? The other option could be to push the data to stdin from which Filebeat is reading.

Also feel free to open a feature request.

owulff · November 24, 2015, 9:38pm

Was busy with testing RC2, nightly builds recently and can now come back to this question...

I was inspired by the nginx beat which uses the same approach to pull the information. Maybe it's more an architecture decision. IMHO, well defined monitoring interfaces to pull information like http, jmx should be handled by a beat implementation itself. Then you can avoid an additional daemon running on the system which feeds the data into your proposed generic beat with an http input.

Thanks
Oli

ruflin · November 25, 2015, 7:48am

@owulff Thanks for getting back and sharing your thoughts. I agree we should have as few "processes" running on the host as possible. Having a beat for this quite nice. If this is a common need it could also be that perhaps someone from the community picks it up.

christiangalsterer · December 6, 2015, 9:21pm

I was looking also for a way to monitor our server farms with a beat. As there was no beat available I created the new general purpose Httpbeat. Httpbeat can call any HTTP server and send the result to the various beat outputs, e.g. logstash, elasticearch.
I'm currently using it to call Jolokia and Apache Stats.

The beat is available at https://github.com/christiangalsterer/httpbeat.
The current release is 1.0.0-beta1 and normal get/post works. There are no ready-made packages available (currently working on this) but it can be easily compiled with go.

In case of problems/questions/ideas please feel free to open an issue on github.

ruflin · December 7, 2015, 10:29am

Very cool to see one more beat. We have a public list of community beats here: https://www.elastic.co/guide/en/beats/libbeat/current/community-beats.html

We would love to also add your beat to our community list. This is list is based on the following file:

github.com

elastic/beats/blob/master/libbeat/docs/communitybeats.asciidoc

//////////////////////////////////////////////////////////////////////////
//// This content appears in both the Beats Platform Reference and the
//// Beats Developer Guide.
//////////////////////////////////////////////////////////////////////////

[[community-beats]]
== Community beats

The open source community has been hard at work developing new Beats. You can check
out some of them here.

Have a question about a community Beat? You can post questions and discuss issues in the
https://discuss.elastic.co/c/beats/community-beats[Community Beats] category of the Beats discussion forum.

Have you created a Beat that's not listed? If so, add the name and description of your Beat to the source document for
https://github.com/elastic/beats/blob/master/libbeat/docs/communitybeats.asciidoc[Community Beats] and https://help.github.com/articles/using-pull-requests[open a pull request] in the https://github.com/elastic/beats[Beats GitHub repository] to get your change merged. When you're ready, go ahead and https://discuss.elastic.co/c/annoucements[announce] your new Beat in the Elastic
discussion forum.

NOTE: Elastic provides no warranty or support for community-sourced Beats.

This file has been truncated. show original

Feel free to open a PR if you like to add it.

christiangalsterer · December 7, 2015, 6:30pm

Httpbeat is already listed

ruflin · December 7, 2015, 7:07pm

@christiangalsterer Seems like it was too early today in the morning Sorry about that.