Jmx

Elastic Stack Elasticsearch
1

Hello,

JMX is a common way to monitor a lot of Java Apps and a lot a monitor apps
have JMX connector. I read on this group that Shay does not recommend JMX
for monitoring ES. Stats API is prefered. Is there any reason for that ?

I know JMX has a cost. But with low frequency queries, it become
unnoticeable.

Regards,

Nicolas

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

2

Hi,

Not only is JMX not recommended, but it's been completely removed from ES
as far as I understand. Our SPM tool for monitoring ES uses the stats API.

Otis

ELASTICSEARCH Performance Monitoring - Sematext Monitoring | Infrastructure Monitoring Servicehttp://sematext.com/spm/index.html
Search Analytics - Cloud Monitoring Tools & Services | Sematexthttp://sematext.com/search-analytics/index.html

On Friday, May 24, 2013 5:23:38 AM UTC-4, Nicolas Labrot wrote:

Hello,

JMX is a common way to monitor a lot of Java Apps and a lot a monitor apps
have JMX connector. I read on this group that Shay does not recommend JMX
for monitoring ES. Stats API is prefered. Is there any reason for that ?

I know JMX has a cost. But with low frequency queries, it become
unnoticeable.

Regards,

Nicolas

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

3

But any idea why JMX is not recommanded?

Your SPM tool seems great, but ops will see it as another tool to monitor
just one app.

Le vendredi 24 mai 2013 18:15:27 UTC+2, Otis Gospodnetic a écrit :

Hi,

Not only is JMX not recommended, but it's been completely removed from ES
as far as I understand. Our SPM tool for monitoring ES uses the stats API.

Otis

ELASTICSEARCH Performance Monitoring - Sematext Monitoring | Infrastructure Monitoring Servicehttp://sematext.com/spm/index.html
Search Analytics - Cloud Monitoring Tools & Services | Sematexthttp://sematext.com/search-analytics/index.html

On Friday, May 24, 2013 5:23:38 AM UTC-4, Nicolas Labrot wrote:

Hello,

JMX is a common way to monitor a lot of Java Apps and a lot a monitor
apps have JMX connector. I read on this group that Shay does not recommend
JMX for monitoring ES. Stats API is prefered. Is there any reason for that ?

I know JMX has a cost. But with low frequency queries, it become
unnoticeable.

Regards,

Nicolas

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

4

The JSON metrics API exported by Elasticsearch is far simpler to get data
out of than JMX for a lot of monitoring systems. Talking to JMX generally
requires a Java based agent for the metrics gathering system or using
something like jmxtrans to get the data out.

Regardless, Elasticsearch has only removed support for creating its own JMX
connector. You can still use the JVM command line flags like this

-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.port=9400
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.authenticate=false

To have the JVM create a remote connector for JMX access. If you add these
to the JAVA_OPTS environment variable before invoking Elasticsearch they'll
get automatically added to the Elasticsearch command line. It's essentially
equivalent to what you got before with jmx.create_connector: true

Cheers,
Dan

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

5

Hi,

On Fri, May 24, 2013 at 1:04 PM, Nicolas Labrot nithril@gmail.com wrote:

But any idea why JMX is not recommanded?

I suspect it was removed because it meant more maintenance work, and
metrics can be retrieved via HTTP API.

Your SPM tool seems great, but ops will see it as another tool to monitor
just one app.

Ah, but that's not really true. SPM monitors not just Elasticsearch,
but also Hadoop, HBase, Solr, Sensei, JVM, and servers.

Tell me what else you want SPM to monitor?

Thanks,
Otis

Solr & Elasticsearch Support

Le vendredi 24 mai 2013 18:15:27 UTC+2, Otis Gospodnetic a écrit :

Hi,

Not only is JMX not recommended, but it's been completely removed from ES
as far as I understand. Our SPM tool for monitoring ES uses the stats API.

Otis

ELASTICSEARCH Performance Monitoring - Sematext Monitoring | Infrastructure Monitoring Service
Search Analytics - Cloud Monitoring Tools & Services | Sematext

On Friday, May 24, 2013 5:23:38 AM UTC-4, Nicolas Labrot wrote:

Hello,

JMX is a common way to monitor a lot of Java Apps and a lot a monitor
apps have JMX connector. I read on this group that Shay does not recommend
JMX for monitoring ES. Stats API is prefered. Is there any reason for that ?

I know JMX has a cost. But with low frequency queries, it become
unnoticeable.

Regards,

Nicolas

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/tUBRE0UotCU/unsubscribe?hl=en-US.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

6

Hello Dan,

Le lundi 27 mai 2013 01:01:05 UTC+2, Dan Everton a écrit :

The JSON metrics API exported by Elasticsearch is far simpler to get data
out of than JMX for a lot of monitoring systems. Talking to JMX generally
requires a Java based agent for the metrics gathering system or using
something like jmxtrans to get the data out.

I agree the JSON API is simpler to query. But this is a specific API which
need custom connector. JMX is de facto a standard for monitoring Java apps.
There are many tools/connectors/monitoring apps that can query mbeans.

Regardless, Elasticsearch has only removed support for creating its own
JMX connector. You can still use the JVM command line flags like this

-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.port=9400
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.authenticate=false

To have the JVM create a remote connector for JMX access. If you add these
to the JAVA_OPTS environment variable before invoking Elasticsearch they'll
get automatically added to the Elasticsearch command line. It's essentially
equivalent to what you got before with jmx.create_connector: true

My message is about ES mbeans which have been removed.

Regards,

Nicolas

Cheers,
Dan

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

7

My message is about ES mbeans which have been removed.

I missed that the actual MBeans had been removed along with the connector.
That is a bit tricky to deal with if you need to stick with JMX and can't
adjust your monitoring tools to consume JSON. One option would be to write
a plugin for Elasticsearch that contains the MBeans for the metrics you
need and just wraps the existing statistics.

Cheers,
Dan

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

8

Le mardi 28 mai 2013 00:55:51 UTC+2, Dan Everton a écrit :

My message is about ES mbeans which have been removed.

I missed that the actual MBeans had been removed along with the connector.
That is a bit tricky to deal with if you need to stick with JMX and can't
adjust your monitoring tools to consume JSON. One option would be to write
a plugin for Elasticsearch that contains the MBeans for the metrics you
need and just wraps the existing statistics.

Absolutely, a plugin should not be so hard to do.

Cheers,
Dan

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

9

Have you started work on an JMX plugin?

Note, the overhead comes form RMI. As long as JMX is not running, it does
not matter, but in the long run, the RMI and GC overhead are significant,
as still noted in Elasticsearch Platform — Find real-time answers at scale | Elastic

There is a glitch which makes a JMX plugin implementation not very handy.
Either you add a static set of MBeans. Or a dynamic one, but for this, you
have to add a JMX MBeans annotation jar to the main ES classpath in
$ES_HOME/lib (JSR 255 like, for an example
GitHub - wjw465150/jmx-annotations: Create and register JMX MBeans by adding some simple annotations to your classes). Otherwise, classes
annotated for being MBeans won't get recognized at ES node start time by
the Guice plugin initiator.

Jörg

On Tuesday, May 28, 2013 9:11:22 AM UTC+2, Nicolas Labrot wrote:

Le mardi 28 mai 2013 00:55:51 UTC+2, Dan Everton a écrit :

My message is about ES mbeans which have been removed.

I missed that the actual MBeans had been removed along with the
connector. That is a bit tricky to deal with if you need to stick with JMX
and can't adjust your monitoring tools to consume JSON. One option would be
to write a plugin for Elasticsearch that contains the MBeans for the
metrics you need and just wraps the existing statistics.

Absolutely, a plugin should not be so hard to do.

Cheers,
Dan

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

10

I guess people know here I am not a fan of JMX, specifically because of the problem to present complex data structures (Ike size per shard on a node) and not being able to easily display stats across nodes (like the size of an index and all its shards across a cluster). also, it does add an annoying overhead.

always thought that stats in Java products tended to be quite useless because of the limiting factors of JMX... (either in interaction or exposing it)

we removed the auto connector support and the few mbeans we had (they didnt really expose that much data). the effort involved in trying to support the breadth of stats ES supports would not have even been challenging using JMX, but basically impossible...

On Fri, May 31, 2013 at 4:49 PM, Jörg Prante joergprante@gmail.com
wrote:

Have you started work on an JMX plugin?
Note, the overhead comes form RMI. As long as JMX is not running, it does
not matter, but in the long run, the RMI and GC overhead are significant,
as still noted in Elasticsearch Platform — Find real-time answers at scale | Elastic
There is a glitch which makes a JMX plugin implementation not very handy.
Either you add a static set of MBeans. Or a dynamic one, but for this, you
have to add a JMX MBeans annotation jar to the main ES classpath in
$ES_HOME/lib (JSR 255 like, for an example
GitHub - wjw465150/jmx-annotations: Create and register JMX MBeans by adding some simple annotations to your classes). Otherwise, classes
annotated for being MBeans won't get recognized at ES node start time by
the Guice plugin initiator.
Jörg
On Tuesday, May 28, 2013 9:11:22 AM UTC+2, Nicolas Labrot wrote:

Le mardi 28 mai 2013 00:55:51 UTC+2, Dan Everton a écrit :

My message is about ES mbeans which have been removed.

I missed that the actual MBeans had been removed along with the
connector. That is a bit tricky to deal with if you need to stick with JMX
and can't adjust your monitoring tools to consume JSON. One option would be
to write a plugin for Elasticsearch that contains the MBeans for the
metrics you need and just wraps the existing statistics.

Absolutely, a plugin should not be so hard to do.

Cheers,
Dan

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

11

Thanks for your reply.

@Jörg: I will choose the shortest path and JMX seems to be actually not
the shortest one. A logstash input plugin may be more appropriate

On Sat, Jun 1, 2013 at 3:14 AM, Shay Banon kimchy@gmail.com wrote:

I guess people know here I am not a fan of JMX, specifically because of
the problem to present complex data structures (Ike size per shard on a
node) and not being able to easily display stats across nodes (like the
size of an index and all its shards across a cluster). also, it does add an
annoying overhead.

always thought that stats in Java products tended to be quite useless
because of the limiting factors of JMX... (either in interaction or
exposing it)

we removed the auto connector support and the few mbeans we had (they
didnt really expose that much data). the effort involved in trying to
support the breadth of stats ES supports would not have even been
challenging using JMX, but basically impossible...

On Fri, May 31, 2013 at 4:49 PM, Jörg Prante joergprante@gmail.comwrote:

Have you started work on an JMX plugin?

Note, the overhead comes form RMI. As long as JMX is not running, it does
not matter, but in the long run, the RMI and GC overhead are significant,
as still noted in
Elasticsearch Platform — Find real-time answers at scale | Elastic

There is a glitch which makes a JMX plugin implementation not very handy.
Either you add a static set of MBeans. Or a dynamic one, but for this, you
have to add a JMX MBeans annotation jar to the main ES classpath in
$ES_HOME/lib (JSR 255 like, for an example
GitHub - wjw465150/jmx-annotations: Create and register JMX MBeans by adding some simple annotations to your classes). Otherwise, classes
annotated for being MBeans won't get recognized at ES node start time by
the Guice plugin initiator.

Jörg

On Tuesday, May 28, 2013 9:11:22 AM UTC+2, Nicolas Labrot wrote:

Le mardi 28 mai 2013 00:55:51 UTC+2, Dan Everton a écrit :

My message is about ES mbeans which have been removed.

I missed that the actual MBeans had been removed along with the
connector. That is a bit tricky to deal with if you need to stick with JMX
and can't adjust your monitoring tools to consume JSON. One option would be
to write a plugin for Elasticsearch that contains the MBeans for the
metrics you need and just wraps the existing statistics.

Absolutely, a plugin should not be so hard to do.

Cheers,
Dan

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

12

I have done a test using last snapshot 1.x but that does not work

Add to the elasticsearch script -->

JAVA_OPTS="-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.port=9400
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.authenticate=false"

After starting jconsole and connect to elasticsearch bootstrap, then no ES
MBeans appear

I see from the code that we have commented those lines in Bootstrap. This
is perhaps the explanation

private static Tuple<Settings, Environment> setupJmx(Tuple<Settings,

Environment> tuple) {
// We disable JMX on by default, since we don't really want the
overhead of RMI (and RMI GC...)
// if (tuple.v1().get(JmxService.SettingsConstants.CREATE_CONNECTOR)
== null) {
// // automatically create the connector if we are bootstrapping
// Settings updated =
settingsBuilder().put(tuple.v1()).put(JmxService.SettingsConstants.CREATE_CONNECTOR,
true).build();
// tuple = new Tuple<Settings, Environment>(updated, tuple.v2());
// }
return tuple;
}

On Friday, May 24, 2013 11:23:38 AM UTC+2, Nicolas Labrot wrote:

Hello,

JMX is a common way to monitor a lot of Java Apps and a lot a monitor apps
have JMX connector. I read on this group that Shay does not recommend JMX
for monitoring ES. Stats API is prefered. Is there any reason for that ?

I know JMX has a cost. But with low frequency queries, it become
unnoticeable.

Regards,

Nicolas

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.