Performance metrics reporting tools for ES


(T Vinod Gupta) #1

i want to be able to measure the health of my ES deployment by looking at
various key counters/metrics like throughput, latency, load, disk IOs, etc.
that way i have visibility on how things are progressing..

are there any recommendations or does ES have an API for the same?

i see some solutions when i google for it -

  1. http://sematext.com/spm/elasticsearch-performance-monitoring/index.html
  2. http://manpages.ubuntu.com/manpages/oneiric/man1/pmdaelasticsearch.1.html

are these particularly good enough?

thanks


(Radu Gheorghe) #2

If you want to look at how these metrics look like (without alerts, for
example), I've used BigDesk and I liked it:

It seems to be the popular choice.

On Thursday, July 5, 2012 4:21:01 AM UTC+3, T Vinod Gupta wrote:

i want to be able to measure the health of my ES deployment by looking at
various key counters/metrics like throughput, latency, load, disk IOs, etc.
that way i have visibility on how things are progressing..

are there any recommendations or does ES have an API for the same?

i see some solutions when i google for it -

  1. http://sematext.com/spm/elasticsearch-performance-monitoring/index.html

http://manpages.ubuntu.com/manpages/oneiric/man1/pmdaelasticsearch.1.html

are these particularly good enough?

thanks


(Otis Gospodnetić) #3

Hello,

For what it's worth we use SPM for ElasticSearch a LOT in all our
ElasticSearch work/engagements with clients. Radu mentioned Alerts, and
SPM has them in the current release, though we are keeping them limited to
a few test accounts in the backend for now. I think BigDesk is handy for
"ad-hoc" looking at ES, while SPM is handy if you want to continuously
monitor ES (and not just ES), if you want to be alerted when there are
issues, if you want to be able to look at historical data, etc.

Otis

Search Analytics - http://sematext.com/search-analytics/index.html
Scalable Performance Monitoring - http://sematext.com/spm/index.html

On Wednesday, July 4, 2012 9:21:01 PM UTC-4, T Vinod Gupta wrote:

i want to be able to measure the health of my ES deployment by looking at
various key counters/metrics like throughput, latency, load, disk IOs, etc.
that way i have visibility on how things are progressing..

are there any recommendations or does ES have an API for the same?

i see some solutions when i google for it -

  1. http://sematext.com/spm/elasticsearch-performance-monitoring/index.html

http://manpages.ubuntu.com/manpages/oneiric/man1/pmdaelasticsearch.1.html

are these particularly good enough?

thanks


(T Vinod Gupta) #4

Hi Otis,
this looks like a really cool tool. i was excited and tried to install it.
somehow it didnt work for me..
when i installed and made the changes to elasticsearch.in.sh file,
elasticsearch won't start. it starts when i revert the changes made to that
file. after installing spm, i just added this line to
elasticsearch.in.shat the very end -

JAVA_OPTS="$JAVA_OPTS -Dcom.sun.management.jmxremote
-javaagent:/spm/spm-monitor/lib/spm-monitor-es-1.5.0-withdeps.jar=/spm/spm-monitor/conf/spm-monitor-config--default.xml"

i start elasticsearch by just doing "./bin/elasticsearch". am i missing
something?

thanks

On Thu, Jul 5, 2012 at 10:12 AM, Otis Gospodnetic <
otis.gospodnetic@gmail.com> wrote:

Hello,

For what it's worth we use SPM for ElasticSearch a LOT in all our
ElasticSearch work/engagements with clients. Radu mentioned Alerts, and
SPM has them in the current release, though we are keeping them limited to
a few test accounts in the backend for now. I think BigDesk is handy for
"ad-hoc" looking at ES, while SPM is handy if you want to continuously
monitor ES (and not just ES), if you want to be alerted when there are
issues, if you want to be able to look at historical data, etc.

Otis

Search Analytics - http://sematext.com/search-analytics/index.html
Scalable Performance Monitoring - http://sematext.com/spm/index.html

On Wednesday, July 4, 2012 9:21:01 PM UTC-4, T Vinod Gupta wrote:

i want to be able to measure the health of my ES deployment by looking at
various key counters/metrics like throughput, latency, load, disk IOs, etc.
that way i have visibility on how things are progressing..

are there any recommendations or does ES have an API for the same?

i see some solutions when i google for it -

  1. http://sematext.com/spm/elasticsearch-performance-
    monitoring/index.htmlhttp://sematext.com/spm/elasticsearch-performance-monitoring/index.html
  2. http://manpages.ubuntu.com/manpages/oneiric/man1/
    pmdaelasticsearch.1.htmlhttp://manpages.ubuntu.com/manpages/oneiric/man1/pmdaelasticsearch.1.html

are these particularly good enough?

thanks


(Otis Gospodnetić) #5

Hi Vinod,

That looks correct. Why don't we switch over to this communication
channel, though, so we don't bother everyone here:

Otis

Search Analytics - http://sematext.com/search-analytics/index.html
Scalable Performance Monitoring - http://sematext.com/spm/index.html

On Sunday, July 8, 2012 10:22:53 PM UTC-4, T Vinod Gupta wrote:

Hi Otis,
this looks like a really cool tool. i was excited and tried to install it.
somehow it didnt work for me..
when i installed and made the changes to elasticsearch.in.sh file,
elasticsearch won't start. it starts when i revert the changes made to that
file. after installing spm, i just added this line to elasticsearch.in.shat the very end -

JAVA_OPTS="$JAVA_OPTS -Dcom.sun.management.jmxremote
-javaagent:/spm/spm-monitor/lib/spm-monitor-es-1.5.0-withdeps.jar=/spm/spm-monitor/conf/spm-monitor-config--default.xml"

i start elasticsearch by just doing "./bin/elasticsearch". am i missing
something?

thanks

On Thu, Jul 5, 2012 at 10:12 AM, Otis Gospodnetic <
otis.gospodnetic@gmail.com> wrote:

Hello,

For what it's worth we use SPM for ElasticSearch a LOT in all our
ElasticSearch work/engagements with clients. Radu mentioned Alerts, and
SPM has them in the current release, though we are keeping them limited to
a few test accounts in the backend for now. I think BigDesk is handy for
"ad-hoc" looking at ES, while SPM is handy if you want to continuously
monitor ES (and not just ES), if you want to be alerted when there are
issues, if you want to be able to look at historical data, etc.

Otis

Search Analytics - http://sematext.com/search-analytics/index.html
Scalable Performance Monitoring - http://sematext.com/spm/index.html

On Wednesday, July 4, 2012 9:21:01 PM UTC-4, T Vinod Gupta wrote:

i want to be able to measure the health of my ES deployment by looking
at various key counters/metrics like throughput, latency, load, disk IOs,
etc. that way i have visibility on how things are progressing..

are there any recommendations or does ES have an API for the same?

i see some solutions when i google for it -

  1. http://sematext.com/spm/elasticsearch-performance-
    monitoring/index.htmlhttp://sematext.com/spm/elasticsearch-performance-monitoring/index.html
  2. http://manpages.ubuntu.com/manpages/oneiric/man1/
    pmdaelasticsearch.1.htmlhttp://manpages.ubuntu.com/manpages/oneiric/man1/pmdaelasticsearch.1.html

are these particularly good enough?

thanks


(Brad Lhotsky) #6

I've been fighting with some problems on ES 0.18.x (production) and ES
0.19.x in my dev environment and came up with a home brew solution for
performance monitoring ElasticSearch:

http://divisionbyzero.net/article/2012/07/09/silly-graphite-trick.html

I like BigDesk for a heads up view, but Graphite is king for being able
compare data sets..

On Thursday, July 5, 2012 3:21:01 AM UTC+2, T Vinod Gupta wrote:

i want to be able to measure the health of my ES deployment by looking at
various key counters/metrics like throughput, latency, load, disk IOs, etc.
that way i have visibility on how things are progressing..

are there any recommendations or does ES have an API for the same?

i see some solutions when i google for it -

  1. http://sematext.com/spm/elasticsearch-performance-monitoring/index.html

http://manpages.ubuntu.com/manpages/oneiric/man1/pmdaelasticsearch.1.html

are these particularly good enough?

thanks


(Lukáš Vlček) #7

Hi,

speaking about bigdesk Otis is right, people can find it handy for quick
checking of the cluster (for now namely Node Stats) - that is what it was
build for. But for now it lacks features that would make it a solid
production tool like ability to store large amount of data into some
datastore, alerts and customizations. Personally, I am/was using bigdesk
especially when I was experimenting with elasticsearch as it allowed me to
quickly realize how to better plan resources for production deployment.
More then monitoring tool bigdesk is like an exploration tool.

You can also try to check elasticsearch-paramedic [1] by Karmi. But again,
IMO it lacks the same features like bigdesk that would make it a solid
production tool, though it brings some cool charts and some fancy signals
for red cluster has been implemented there recently! :slight_smile:

Regards,
Lukas

[1] https://github.com/karmi/elasticsearch-paramedic

On Tue, Jul 10, 2012 at 12:31 AM, Brad Lhotsky brad.lhotsky@gmail.comwrote:

I've been fighting with some problems on ES 0.18.x (production) and ES
0.19.x in my dev environment and came up with a home brew solution for
performance monitoring ElasticSearch:

http://divisionbyzero.net/article/2012/07/09/silly-graphite-trick.html

I like BigDesk for a heads up view, but Graphite is king for being able
compare data sets..

On Thursday, July 5, 2012 3:21:01 AM UTC+2, T Vinod Gupta wrote:

i want to be able to measure the health of my ES deployment by looking at
various key counters/metrics like throughput, latency, load, disk IOs, etc.
that way i have visibility on how things are progressing..

are there any recommendations or does ES have an API for the same?

i see some solutions when i google for it -

  1. http://sematext.com/spm/elasticsearch-performance-
    monitoring/index.htmlhttp://sematext.com/spm/elasticsearch-performance-monitoring/index.html
  2. http://manpages.ubuntu.com/manpages/oneiric/man1/
    pmdaelasticsearch.1.htmlhttp://manpages.ubuntu.com/manpages/oneiric/man1/pmdaelasticsearch.1.html

are these particularly good enough?

thanks


(Lukáš Vlček) #8

Hi,

just found this interesting blog post
http://divisionbyzero.net/article/2012/07/09/silly-graphite-trick.html
I think it might be useful to post it here for inspiration.

Regards,
Lukas

On Tue, Jul 10, 2012 at 6:53 AM, Lukáš Vlček lukas.vlcek@gmail.com wrote:

Hi,

speaking about bigdesk Otis is right, people can find it handy for quick
checking of the cluster (for now namely Node Stats) - that is what it was
build for. But for now it lacks features that would make it a solid
production tool like ability to store large amount of data into some
datastore, alerts and customizations. Personally, I am/was using bigdesk
especially when I was experimenting with elasticsearch as it allowed me to
quickly realize how to better plan resources for production deployment.
More then monitoring tool bigdesk is like an exploration tool.

You can also try to check elasticsearch-paramedic [1] by Karmi. But again,
IMO it lacks the same features like bigdesk that would make it a solid
production tool, though it brings some cool charts and some fancy signals
for red cluster has been implemented there recently! :slight_smile:

Regards,
Lukas

[1] https://github.com/karmi/elasticsearch-paramedic

On Tue, Jul 10, 2012 at 12:31 AM, Brad Lhotsky brad.lhotsky@gmail.comwrote:

I've been fighting with some problems on ES 0.18.x (production) and ES
0.19.x in my dev environment and came up with a home brew solution for
performance monitoring ElasticSearch:

http://divisionbyzero.net/article/2012/07/09/silly-graphite-trick.html

I like BigDesk for a heads up view, but Graphite is king for being able
compare data sets..

On Thursday, July 5, 2012 3:21:01 AM UTC+2, T Vinod Gupta wrote:

i want to be able to measure the health of my ES deployment by looking
at various key counters/metrics like throughput, latency, load, disk IOs,
etc. that way i have visibility on how things are progressing..

are there any recommendations or does ES have an API for the same?

i see some solutions when i google for it -

  1. http://sematext.com/spm/elasticsearch-performance-
    monitoring/index.htmlhttp://sematext.com/spm/elasticsearch-performance-monitoring/index.html
  2. http://manpages.ubuntu.com/manpages/oneiric/man1/
    pmdaelasticsearch.1.htmlhttp://manpages.ubuntu.com/manpages/oneiric/man1/pmdaelasticsearch.1.html

are these particularly good enough?

thanks


(system) #9