Can we use topbeat for monitoring JVM metrics like heap memory, active threads, garbage collection etc. ?
No, it cannot. There is no Beat for JVM monitoring at the current time. This was discuss a bit here:
You can give Httpbeat together with Jolokia a try. Httpbeat is a beat which can call any HTTP endpoint and Jolokia is a HTTP/JSON bridge for JMX. I'm currently using this combination to monitor a set of 30+ servers monitoring common metrics like connections, memory usage.
Disclaimer: I'm the owner of Httpbeat.
Thanks @christiangalsterer. That is interesting. I will give it a try. we have 100+ servers to monitor.
@christiangalsterer, I'm also facing a similar issue. Basically I want to monitor the metrics of a spark cluster. I have the metrics of the master on http://localhost:8081/metrics/master/json/path and those of the workers on http://localhost:4041/metrics/json/, http://localhost:4042/metrics/json/ http://localhost:4042/metrics/json/ and so on.
With this configure please can you help me with some steps or some documentation on how to get those metrics with httpbeat and jolokia.
Thanks.
Hi can find the httpbeat here along with same examples how to query an HTTP endpoint.
E.g. you can point to http://localhost:8081/metrics/master/json/path to query the metrics of your spark master and send those data then to elastic search and/or logstash.
For the document_type I suggest that you use a value of "spark" or something similar.
Thanks for the reply,
Sorry for the delay, I had something more urgent to address. I have done that configuration but the data that I want come in the field body,
"body" : "{\"version\":\"3.0.0\",\"gauges\":{\"jvm.PS-MarkSweep.count\":{\"value\":1},\"jvm.PS-MarkSweep.time\":{\"value\":20},\"jvm.PS-Scavenge.count\":{\"value\":3},\"jvm.PS-Scavenge.time\":{\"value\":25},\"jvm.heap.committed\":{\"value\":342360064},\"jvm.heap.init\":{\"value\":526385152},\"jvm.heap.max\":{\"value\":954728448},\"jvm.heap.usage\":{\"value\":0.1372370628260571},\"jvm.heap.used\":{\"value\":131024128},\"jvm.non-heap.committed\":{\"value\":41353216},\"jvm.non-heap.init\":{\"value\":2555904},\"jvm.non-heap.max\":{\"value\":-1},\"jvm.non-heap.usage\":{\"value\":-4.0685104E7},\"jvm.non-heap.used\":{\"value\":40685104},\"jvm.pools.Code-Cache.committed\":{\"value\":7012352},\"jvm.pools.Code-Cache.init\":{\"value\":2555904},\"jvm.pools.Code-Cache.max\":{\"value\":251658240},\"jvm.pools.Code-Cache.usage\":{\"value\":0.02767003377278646},\"jvm.pools.Code-Cache.used\":{\"value\":6964096},\"jvm.pools.Compressed-Class-Space.committed\":{\"value\":4194304},\"jvm.pools.Compressed-Class-Space.init\":{\"value\":0},\"jvm.pools.Compressed-Class-Space.max\":{\"value\":1073741824},\"jvm.pools.Compressed-Class-Space.usage\":{\"value\":0.0038136616349220276},\"jvm.pools.Compressed-Class-Space.used\":{\"value\":4094888},\"jvm.pools.Metaspace.committed\":{\"value\":30146560},\"jvm.pools.Metaspace.init\":{\"value\":0},\"jvm.pools.Metaspace.max\":{\"value\":-1},\"jvm.pools.Metaspace.usage\":{\"value\":0.9827830439028533},\"jvm.pools.Metaspace.used\":{\"value\":29627528},\"jvm.pools.PS-Eden-Space.committed\":{\"value\":132120576},\"jvm.pools.PS-Eden-Space.init\":{\"value\":132120576},\"jvm.pools.PS-Eden-Space.max\":{\"value\":314572800},\"jvm.pools.PS-Eden-Space.usage\":{\"value\":0.263955561319987},\"jvm.pools.PS-Eden-Space.used\":{\"value\":83033240},\"jvm.pools.PS-Old-Gen.committed\":{\"value\":188743680},\"jvm.pools.PS-Old-Gen.init\":{\"value\":351272960},\"jvm.pools.PS-Old-Gen.max\":{\"value\":716177408},\"jvm.pools.PS-Old-Gen.usage\":{\"value\":0.037000530460743045},\"jvm.pools.PS-Old-Gen.used\":{\"value\":26498944},\"jvm.pools.PS-Survivor-Space.committed\":{\"value\":21495808},\"jvm.pools.PS-Survivor-Space.init\":{\"value\":21495808},\"jvm.pools.PS-Survivor-Space.max\":{\"value\":21495808},\"jvm.pools.PS-Survivor-Space.usage\":{\"value\":0.999820244021532},\"jvm.pools.PS-Survivor-Space.used\":{\"value\":21491944},\"jvm.total.committed\":{\"value\":383713280},\"jvm.total.init\":{\"value\":528941056},\"jvm.total.max\":{\"value\":954728447},\"jvm.total.used\":{\"value\":171714072},\"master.aliveWorkers\":{\"value\":1},\"master.apps\":{\"value\":1},\"master.waitingApps\":{\"value\":0},\"master.workers\":{\"value\":1}},\"counters\":{},\"histograms\":{\"CodeGenerator.compilationTime\":{\"count\":0,\"max\":0,\"mean\":0.0,\"min\":0,\"p50\":0.0,\"p75\":0.0,\"p95\":0.0,\"p98\":0.0,\"p99\":0.0,\"p999\":0.0,\"stddev\":0.0},\"CodeGenerator.generatedClassSize\":{\"count\":0,\"max\":0,\"mean\":0.0,\"min\":0,\"p50\":0.0,\"p75\":0.0,\"p95\":0.0,\"p98\":0.0,\"p99\":0.0,\"p999\":0.0,\"stddev\":0.0},\"CodeGenerator.generatedMethodSize\":{\"count\":0,\"max\":0,\"mean\":0.0,\"min\":0,\"p50\":0.0,\"p75\":0.0,\"p95\":0.0,\"p98\":0.0,\"p99\":0.0,\"p999\":0.0,\"stddev\":0.0},\"CodeGenerator.sourceCodeSize\":{\"count\":0,\"max\":0,\"mean\":0.0,\"min\":0,\"p50\":0.0,\"p75\":0.0,\"p95\":0.0,\"p98\":0.0,\"p99\":0.0,\"p999\":0.0,\"stddev\":0.0}},\"meters\":{},\"timers\":{}}",
I was wondering if it is possible to configure httpbeat so that those data come in different fields. That is
{
"version":"3.0.0",
"gauges":{"jvm.PS-MarkSweep.count":{"value":1},
"jvm.PS-MarkSweep.time":{"value":20},
"jvm.PS-Scavenge.count":{"value":3},
"jvm.PS-Scavenge.time":{"value":25},
"jvm.heap.committed":{"value":342360064},
"jvm.heap.init":{"value":526385152},
....
}
Thanks.
Currently it is not possible to change the root field. Instead of sending the data directly to Elasticsearch you can send the data first to Logstash and use one powerful features and plugins to manipulate the payload before forwarding it to Elasticsearch. You can also try to use the ingest node feature of Elasticsearch which provides similar capabilities
but I haven't use the ingest node myself yet.
In case you have jolokia in place, you could also be interested in the jolokia module: https://github.com/elastic/beats/pull/3570
Hi Ruflin,
That module seems to be awesome, is it safe to integrate with ES 5? how can I do it? any repo? or should I clone a git repo? can you provide some steps?
Thanks
Regards
The jolokia module is in the nightly builds (https://beats-nightlies.s3.amazonaws.com/index.html?prefix=metricbeat/) and will be part of 5.4 release. The data structure is compatible with ES 5.