Keeping JMX and REST metrics APIs in sync

Elastic Stack Elasticsearch
1

Hi,

Shay - what are the chances of metrics exposed in JMX getting (and
remaining!) in sync with metrics exposed via REST APIs?
I know you don't like JMX because of the overhead, but JMX is already
there (so already has whatever overhead it adds), it's just that it's
not useful because it doesn't have all metrics.

So if one/you were to bring metrics in JMX up to date, what are the
chances of you making sure to keep that in sync with metrics in REST
APIs? (basically, if new metric becomes available, it should be
exposed via JMX, too)

Thanks,
Otis

2

On Saturday, February 4, 2012 at 12:08 AM, Otis Gospodnetic wrote:

Hi,

Shay - what are the chances of metrics exposed in JMX getting (and
remaining!) in sync with metrics exposed via REST APIs?
I know you don't like JMX because of the overhead, but JMX is already
there (so already has whatever overhead it adds), it's just that it's
not useful because it doesn't have all metrics.

First, the overhead of JMX mainly comes from RMI, and thats only "started" when you start to monitor it remotely (and the annoying GC aspects of it as well, like full GC unless configured properly).

More monitoring systems today have the ability to call a simple REST service and extract the relevant metrics from json. This option is vastly superior to JMX (over RMI) because you can get proper "objects" back and not ping each attribute separately, already uses HTTP so it works well with things like load balancer and security, and the like.

So if one/you were to bring metrics in JMX up to date, what are the
chances of you making sure to keep that in sync with metrics in REST
APIs? (basically, if new metric becomes available, it should be
exposed via JMX, too)

Its a matter of being diligent, and whenever a new stat is added, expose it.

Thanks,
Otis

http://sematext.com/

3

Hi,

On Feb 5, 11:26 am, Shay Banon kim...@gmail.com wrote:

On Saturday, February 4, 2012 at 12:08 AM, Otis Gospodnetic wrote:

Hi,

Shay - what are the chances of metrics exposed in JMX getting (and
remaining!) in sync with metrics exposed via REST APIs?
I know you don't like JMX because of the overhead, but JMX is already
there (so already has whatever overhead it adds), it's just that it's
not useful because it doesn't have all metrics.

First, the overhead of JMX mainly comes from RMI, and thats only "started" when you start to monitor it remotely (and the annoying GC aspects of it as well, like full GC unless configured properly).

Ah, right, lovely RMI. I can't believe that's still in use!
In my particular case, I am/was interested in connected to JMX from
within ES Node's JVM.

More monitoring systems today have the ability to call a simple REST service and extract the relevant metrics from json. This option is vastly superior to JMX (over RMI) because you can get proper "objects" back and not ping each attribute separately, already uses HTTP so it works well with things like load balancer and security, and the like.

Right. We ended up going against HTTP and parsing JSON.

One problem with this approach, however, is that JSON response changed
between some versions, so in order for a monitoring tool to support
multiple ES versions, it has to have version-specific parsers for that
JSON.

Is there an easy work-around or suggested way for dealing with this
that I'm missing?

We've also tried getting to the stats via the Java API, but that has
the same problem - if you are not ES version-aware, you may call a
method that does not exist in the version of ES being monitored.

So if one/you were to bring metrics in JMX up to date, what are the
chances of you making sure to keep that in sync with metrics in REST
APIs? (basically, if new metric becomes available, it should be
exposed via JMX, too)

Its a matter of being diligent, and whenever a new stat is added, expose it.

Right. I'll open an issue for initial REST+JSON/JMX syncing and cross
my fingers and toes in hopes of that remaining in sync in future
versions. Here it is:

JMX and REST+JSON stats should be in sync · Issue #1675 · elastic/elasticsearch · GitHub

I wonder if one could write unit tests that catch situations where
REST stats and JMX stats don't match 100% in terms of coverage...

Thanks,
Otis

4

I suggest you use the Java API to get the stats, if you are running embedded in ES. It can be provided as a plugin, and yes, if those change (though hopefully we nailed to structure now), you can version your plugin based on elasticsearch versions supported (as other plugins do).

On Tuesday, February 7, 2012 at 5:10 AM, Otis Gospodnetic wrote:

Hi,

On Feb 5, 11:26 am, Shay Banon <kim...@gmail.com (http://gmail.com)> wrote:

On Saturday, February 4, 2012 at 12:08 AM, Otis Gospodnetic wrote:

Hi,

Shay - what are the chances of metrics exposed in JMX getting (and
remaining!) in sync with metrics exposed via REST APIs?
I know you don't like JMX because of the overhead, but JMX is already
there (so already has whatever overhead it adds), it's just that it's
not useful because it doesn't have all metrics.

First, the overhead of JMX mainly comes from RMI, and thats only "started" when you start to monitor it remotely (and the annoying GC aspects of it as well, like full GC unless configured properly).

Ah, right, lovely RMI. I can't believe that's still in use!
In my particular case, I am/was interested in connected to JMX from
within ES Node's JVM.

More monitoring systems today have the ability to call a simple REST service and extract the relevant metrics from json. This option is vastly superior to JMX (over RMI) because you can get proper "objects" back and not ping each attribute separately, already uses HTTP so it works well with things like load balancer and security, and the like.

Right. We ended up going against HTTP and parsing JSON.

One problem with this approach, however, is that JSON response changed
between some versions, so in order for a monitoring tool to support
multiple ES versions, it has to have version-specific parsers for that
JSON.

Is there an easy work-around or suggested way for dealing with this
that I'm missing?

We've also tried getting to the stats via the Java API, but that has
the same problem - if you are not ES version-aware, you may call a
method that does not exist in the version of ES being monitored.

So if one/you were to bring metrics in JMX up to date, what are the
chances of you making sure to keep that in sync with metrics in REST
APIs? (basically, if new metric becomes available, it should be
exposed via JMX, too)

Its a matter of being diligent, and whenever a new stat is added, expose it.

Right. I'll open an issue for initial REST+JSON/JMX syncing and cross
my fingers and toes in hopes of that remaining in sync in future
versions. Here it is:

JMX and REST+JSON stats should be in sync · Issue #1675 · elastic/elasticsearch · GitHub

I wonder if one could write unit tests that catch situations where
REST stats and JMX stats don't match 100% in terms of coverage...

Thanks,
Otis

http://sematext.com/

5

Otis,

I think the only way how to get ES version via REST API is request to "/"
point. However, I am not sure if there is any Java API for that.

Shay,

May be it would be useful to add such Java API or more focused REST API.
IMO "/" endpoint is weakly defined, something like
"/_cluster/nodes/_local/version" or "/_nodes/_local/version" would be
better. Or may be even better include version information into Nodes info
API. That way it would be easy to check that all nodes run on the same
version using a single request, which might be a basic troubleshooting
check for example.

Regards,
Lukas

On Tue, Feb 7, 2012 at 11:25 AM, Shay Banon kimchy@gmail.com wrote:

I suggest you use the Java API to get the stats, if you are running
embedded in ES. It can be provided as a plugin, and yes, if those change
(though hopefully we nailed to structure now), you can version your plugin
based on elasticsearch versions supported (as other plugins do).

On Tuesday, February 7, 2012 at 5:10 AM, Otis Gospodnetic wrote:

Hi,

On Feb 5, 11:26 am, Shay Banon kim...@gmail.com wrote:

On Saturday, February 4, 2012 at 12:08 AM, Otis Gospodnetic wrote:

Hi,

Shay - what are the chances of metrics exposed in JMX getting (and
remaining!) in sync with metrics exposed via REST APIs?
I know you don't like JMX because of the overhead, but JMX is already
there (so already has whatever overhead it adds), it's just that it's
not useful because it doesn't have all metrics.

First, the overhead of JMX mainly comes from RMI, and thats only "started"
when you start to monitor it remotely (and the annoying GC aspects of it as
well, like full GC unless configured properly).

Ah, right, lovely RMI. I can't believe that's still in use!
In my particular case, I am/was interested in connected to JMX from
within ES Node's JVM.

More monitoring systems today have the ability to call a simple REST
service and extract the relevant metrics from json. This option is vastly
superior to JMX (over RMI) because you can get proper "objects" back and
not ping each attribute separately, already uses HTTP so it works well
with things like load balancer and security, and the like.

Right. We ended up going against HTTP and parsing JSON.

One problem with this approach, however, is that JSON response changed
between some versions, so in order for a monitoring tool to support
multiple ES versions, it has to have version-specific parsers for that
JSON.

Is there an easy work-around or suggested way for dealing with this
that I'm missing?

We've also tried getting to the stats via the Java API, but that has
the same problem - if you are not ES version-aware, you may call a
method that does not exist in the version of ES being monitored.

So if one/you were to bring metrics in JMX up to date, what are the
chances of you making sure to keep that in sync with metrics in REST
APIs? (basically, if new metric becomes available, it should be
exposed via JMX, too)

Its a matter of being diligent, and whenever a new stat is added, expose
it.

Right. I'll open an issue for initial REST+JSON/JMX syncing and cross
my fingers and toes in hopes of that remaining in sync in future
versions. Here it is:

JMX and REST+JSON stats should be in sync · Issue #1675 · elastic/elasticsearch · GitHub

I wonder if one could write unit tests that catch situations where
REST stats and JMX stats don't match 100% in terms of coverage...

Thanks,
Otis

http://sematext.com/

6

Yeah, +1 for exposing a version explicitly.

Otis

On Feb 7, 8:34 am, Lukáš Vlček lukas.vl...@gmail.com wrote:

Otis,

I think the only way how to get ES version via REST API is request to "/"
point. However, I am not sure if there is any Java API for that.

Shay,

May be it would be useful to add such Java API or more focused REST API.
IMO "/" endpoint is weakly defined, something like
"/_cluster/nodes/_local/version" or "/_nodes/_local/version" would be
better. Or may be even better include version information into Nodes info
API. That way it would be easy to check that all nodes run on the same
version using a single request, which might be a basic troubleshooting
check for example.

Regards,
Lukas

On Tue, Feb 7, 2012 at 11:25 AM, Shay Banon kim...@gmail.com wrote:

I suggest you use the Java API to get the stats, if you are running
embedded in ES. It can be provided as a plugin, and yes, if those change
(though hopefully we nailed to structure now), you can version your plugin
based on elasticsearch versions supported (as other plugins do).

On Tuesday, February 7, 2012 at 5:10 AM, Otis Gospodnetic wrote:

Hi,

On Feb 5, 11:26 am, Shay Banon kim...@gmail.com wrote:

On Saturday, February 4, 2012 at 12:08 AM, Otis Gospodnetic wrote:

Hi,

Shay - what are the chances of metrics exposed in JMX getting (and
remaining!) in sync with metrics exposed via REST APIs?
I know you don't like JMX because of the overhead, but JMX is already
there (so already has whatever overhead it adds), it's just that it's
not useful because it doesn't have all metrics.

First, the overhead of JMX mainly comes from RMI, and thats only "started"
when you start to monitor it remotely (and the annoying GC aspects of it as
well, like full GC unless configured properly).

Ah, right, lovely RMI. I can't believe that's still in use!
In my particular case, I am/was interested in connected to JMX from
within ES Node's JVM.

More monitoring systems today have the ability to call a simple REST
service and extract the relevant metrics from json. This option is vastly
superior to JMX (over RMI) because you can get proper "objects" back and
not ping each attribute separately, already uses HTTP so it works well
with things like load balancer and security, and the like.

Right. We ended up going against HTTP and parsing JSON.

One problem with this approach, however, is that JSON response changed
between some versions, so in order for a monitoring tool to support
multiple ES versions, it has to have version-specific parsers for that
JSON.

Is there an easy work-around or suggested way for dealing with this
that I'm missing?

We've also tried getting to the stats via the Java API, but that has
the same problem - if you are not ES version-aware, you may call a
method that does not exist in the version of ES being monitored.

So if one/you were to bring metrics in JMX up to date, what are the
chances of you making sure to keep that in sync with metrics in REST
APIs? (basically, if new metric becomes available, it should be
exposed via JMX, too)

Its a matter of being diligent, and whenever a new stat is added, expose
it.

Right. I'll open an issue for initial REST+JSON/JMX syncing and cross
my fingers and toes in hopes of that remaining in sync in future
versions. Here it is:

JMX and REST+JSON stats should be in sync · Issue #1675 · elastic/elasticsearch · GitHub

I wonder if one could write unit tests that catch situations where
REST stats and JMX stats don't match 100% in terms of coverage...

Thanks,
Otis

http://sematext.com/

7

The nodes info Java API already includes the version associated with each node. Each NodeInfo element within the NodesInfoResponse include the node, which has the version associated with it.

On Tuesday, February 7, 2012 at 8:08 PM, Otis Gospodnetic wrote:

Yeah, +1 for exposing a version explicitly.

Otis

http://sematext.com/

On Feb 7, 8:34 am, Lukáš Vlček <lukas.vl...@gmail.com (http://gmail.com)> wrote:

Otis,

I think the only way how to get ES version via REST API is request to "/"
point. However, I am not sure if there is any Java API for that.

Shay,

May be it would be useful to add such Java API or more focused REST API.
IMO "/" endpoint is weakly defined, something like
"/_cluster/nodes/_local/version" or "/_nodes/_local/version" would be
better. Or may be even better include version information into Nodes info
API. That way it would be easy to check that all nodes run on the same
version using a single request, which might be a basic troubleshooting
check for example.

Regards,
Lukas

On Tue, Feb 7, 2012 at 11:25 AM, Shay Banon <kim...@gmail.com (http://gmail.com)> wrote:

I suggest you use the Java API to get the stats, if you are running
embedded in ES. It can be provided as a plugin, and yes, if those change
(though hopefully we nailed to structure now), you can version your plugin
based on elasticsearch versions supported (as other plugins do).

On Tuesday, February 7, 2012 at 5:10 AM, Otis Gospodnetic wrote:

Hi,

On Feb 5, 11:26 am, Shay Banon <kim...@gmail.com (http://gmail.com)> wrote:

On Saturday, February 4, 2012 at 12:08 AM, Otis Gospodnetic wrote:

Hi,

Shay - what are the chances of metrics exposed in JMX getting (and
remaining!) in sync with metrics exposed via REST APIs?
I know you don't like JMX because of the overhead, but JMX is already
there (so already has whatever overhead it adds), it's just that it's
not useful because it doesn't have all metrics.

First, the overhead of JMX mainly comes from RMI, and thats only "started"
when you start to monitor it remotely (and the annoying GC aspects of it as
well, like full GC unless configured properly).

Ah, right, lovely RMI. I can't believe that's still in use!
In my particular case, I am/was interested in connected to JMX from
within ES Node's JVM.

More monitoring systems today have the ability to call a simple REST
service and extract the relevant metrics from json. This option is vastly
superior to JMX (over RMI) because you can get proper "objects" back and
not ping each attribute separately, already uses HTTP so it works well
with things like load balancer and security, and the like.

Right. We ended up going against HTTP and parsing JSON.

One problem with this approach, however, is that JSON response changed
between some versions, so in order for a monitoring tool to support
multiple ES versions, it has to have version-specific parsers for that
JSON.

Is there an easy work-around or suggested way for dealing with this
that I'm missing?

We've also tried getting to the stats via the Java API, but that has
the same problem - if you are not ES version-aware, you may call a
method that does not exist in the version of ES being monitored.

So if one/you were to bring metrics in JMX up to date, what are the
chances of you making sure to keep that in sync with metrics in REST
APIs? (basically, if new metric becomes available, it should be
exposed via JMX, too)

Its a matter of being diligent, and whenever a new stat is added, expose
it.

Right. I'll open an issue for initial REST+JSON/JMX syncing and cross
my fingers and toes in hopes of that remaining in sync in future
versions. Here it is:

JMX and REST+JSON stats should be in sync · Issue #1675 · elastic/elasticsearch · GitHub

I wonder if one could write unit tests that catch situations where
REST stats and JMX stats don't match 100% in terms of coverage...

Thanks,
Otis

http://sematext.com/