Monitoring / Management App: Advice Req

Elastic Stack Elasticsearch
1

Hello All,

I have recently embarked on building a Management and Monitoring app for
ElasticSearch. I do have a few questions on what is most feasible, however:

  1. I have decided to use the JAVA API. It is what ES uses internally, so it
    makes the most sense from a long-term perspective.

  2. Not to be used as a plugin: For monitoring clustered configs in a
    real-world scenario (think 'enterprise), I don't see mass adoption of a
    plugin, but a downloadable war would be most likely to be used.

  3. I could not find anywhere if ES has capabilities to persist usage
    information. Something I would like to add to the app is a way to view
    Analytics and usage information.

Feel free to share your thoughts and feedback on my approach.

Regards,
Roy Russo
http://www.linkedin.com/in/royrusso

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

2

Hi Roy,

Not sure if you saw them, but there are:

Are you looking for something that none of the above tools provides?

Otis

On Thursday, May 2, 2013 11:14:53 AM UTC-4, Roy Russo wrote:

Hello All,

I have recently embarked on building a Management and Monitoring app for
Elasticsearch. I do have a few questions on what is most feasible, however:

  1. I have decided to use the JAVA API. It is what ES uses internally, so
    it makes the most sense from a long-term perspective.

  2. Not to be used as a plugin: For monitoring clustered configs in a
    real-world scenario (think 'enterprise), I don't see mass adoption of a
    plugin, but a downloadable war would be most likely to be used.

  3. I could not find anywhere if ES has capabilities to persist usage
    information. Something I would like to add to the app is a way to view
    Analytics and usage information.

Feel free to share your thoughts and feedback on my approach.

Regards,
Roy Russo
http://www.linkedin.com/in/royrusso

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

3

Thanks Otis. I did see those, but I'm looking to build something a bit more
full-featured that can one day use persistence for historic analysis,
making changes/remembering the cluster config, etc...

Other reasons:

  1. Most common (enterprise?) infrastructure and cloud monitoring apps are
    not normally on the actual running instance of the application one wants to
    monitor/manage. Sometimes those machines are not accessible via port 80 and
    it also risks adding overhead on the instance being monitored. So a
    hub-spoke topography for monitoring is what I prefer for scalability
    reasons.
  2. This is a theoretical, but I feel it is a heavier burden to offload all
    of the logic and data to the client. Instead of atomic calls that can be
    filtered, you risk ending up with "too much" data in the client memory and
    flowing through the pipes.
  3. It is easier to add the capability for a server-side app to be used in a
    mutli-tenant model, ie. offering monitoring/management solutions to ES
    users.

Thoughts?

On Thursday, May 2, 2013 11:14:53 AM UTC-4, Roy Russo wrote:

Hello All,

I have recently embarked on building a Management and Monitoring app for
Elasticsearch. I do have a few questions on what is most feasible, however:

  1. I have decided to use the JAVA API. It is what ES uses internally, so
    it makes the most sense from a long-term perspective.

  2. Not to be used as a plugin: For monitoring clustered configs in a
    real-world scenario (think 'enterprise), I don't see mass adoption of a
    plugin, but a downloadable war would be most likely to be used.

  3. I could not find anywhere if ES has capabilities to persist usage
    information. Something I would like to add to the app is a way to view
    Analytics and usage information.

Feel free to share your thoughts and feedback on my approach.

Regards,
Roy Russo
http://www.linkedin.com/in/royrusso

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

4

Hi,

Not sure I follow.
Sure, you can run a remote agent and pull metrics out of ES. But you may
not be able to get to server metrics, for example. So you need a local
agent. Re too much data - up to the agent what it gets, how often, whether
it compresses, etc.

I won't speak for ES head and BigDesk, but what you are describing doesn't
sound any different than SPM for example. Or how people set up Ganglia. Or
... but maybe I'm missing something? :slight_smile:

Otis

Search Analytics - Cloud Monitoring Tools & Services | Sematext
ELASTICSEARCH Performance Monitoring - Sematext Monitoring | Infrastructure Monitoring Service

On Thursday, May 2, 2013 2:25:45 PM UTC-4, Roy Russo wrote:

Thanks Otis. I did see those, but I'm looking to build something a bit
more full-featured that can one day use persistence for historic analysis,
making changes/remembering the cluster config, etc...

Other reasons:

  1. Most common (enterprise?) infrastructure and cloud monitoring apps are
    not normally on the actual running instance of the application one wants to
    monitor/manage. Sometimes those machines are not accessible via port 80 and
    it also risks adding overhead on the instance being monitored. So a
    hub-spoke topography for monitoring is what I prefer for scalability
    reasons.
  2. This is a theoretical, but I feel it is a heavier burden to offload all
    of the logic and data to the client. Instead of atomic calls that can be
    filtered, you risk ending up with "too much" data in the client memory and
    flowing through the pipes.
  3. It is easier to add the capability for a server-side app to be used in
    a mutli-tenant model, ie. offering monitoring/management solutions to ES
    users.

Thoughts?

On Thursday, May 2, 2013 11:14:53 AM UTC-4, Roy Russo wrote:

Hello All,

I have recently embarked on building a Management and Monitoring app for
Elasticsearch. I do have a few questions on what is most feasible, however:

  1. I have decided to use the JAVA API. It is what ES uses internally, so
    it makes the most sense from a long-term perspective.

  2. Not to be used as a plugin: For monitoring clustered configs in a
    real-world scenario (think 'enterprise), I don't see mass adoption of a
    plugin, but a downloadable war would be most likely to be used.

  3. I could not find anywhere if ES has capabilities to persist usage
    information. Something I would like to add to the app is a way to view
    Analytics and usage information.

Feel free to share your thoughts and feedback on my approach.

Regards,
Roy Russo
http://www.linkedin.com/in/royrusso

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

5

Hi Otis,

1/ It looks like the TransportClient will return all of the data bits that
are exposed via REST - including jvm and system metrics, so there is no
need to drop an agent on the server. I'm testing this locally now and seems
to work.

2/ I looked at SPM just now. I love the ease of registration - nice job.
The only difference I would say here is that there is no need to deploy
anything on the server.

This system would be much like Ganglia - hopefully easier on the eyes. The
one downside I see is that is the development team makes
reverse-incompatible changes, my code will fail miserably, as I bundle the
LATEST in my app, but I guess the same would go for any changes in the REST
responses.

On Thursday, May 2, 2013 3:04:53 PM UTC-4, Otis Gospodnetic wrote:

Hi,

Not sure I follow.
Sure, you can run a remote agent and pull metrics out of ES. But you may
not be able to get to server metrics, for example. So you need a local
agent. Re too much data - up to the agent what it gets, how often, whether
it compresses, etc.

I won't speak for ES head and BigDesk, but what you are describing doesn't
sound any different than SPM for example. Or how people set up Ganglia. Or
... but maybe I'm missing something? :slight_smile:

Otis

Search Analytics - Cloud Monitoring Tools & Services | Sematext
ELASTICSEARCH Performance Monitoring - Sematext Monitoring | Infrastructure Monitoring Service

On Thursday, May 2, 2013 2:25:45 PM UTC-4, Roy Russo wrote:

Thanks Otis. I did see those, but I'm looking to build something a bit
more full-featured that can one day use persistence for historic analysis,
making changes/remembering the cluster config, etc...

Other reasons:

  1. Most common (enterprise?) infrastructure and cloud monitoring apps are
    not normally on the actual running instance of the application one wants to
    monitor/manage. Sometimes those machines are not accessible via port 80 and
    it also risks adding overhead on the instance being monitored. So a
    hub-spoke topography for monitoring is what I prefer for scalability
    reasons.
  2. This is a theoretical, but I feel it is a heavier burden to offload
    all of the logic and data to the client. Instead of atomic calls that can
    be filtered, you risk ending up with "too much" data in the client memory
    and flowing through the pipes.
  3. It is easier to add the capability for a server-side app to be used in
    a mutli-tenant model, ie. offering monitoring/management solutions to ES
    users.

Thoughts?

On Thursday, May 2, 2013 11:14:53 AM UTC-4, Roy Russo wrote:

Hello All,

I have recently embarked on building a Management and Monitoring app for
Elasticsearch. I do have a few questions on what is most feasible, however:

  1. I have decided to use the JAVA API. It is what ES uses internally, so
    it makes the most sense from a long-term perspective.

  2. Not to be used as a plugin: For monitoring clustered configs in a
    real-world scenario (think 'enterprise), I don't see mass adoption of a
    plugin, but a downloadable war would be most likely to be used.

  3. I could not find anywhere if ES has capabilities to persist usage
    information. Something I would like to add to the app is a way to view
    Analytics and usage information.

Feel free to share your thoughts and feedback on my approach.

Regards,
Roy Russo
http://www.linkedin.com/in/royrusso

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

6

I will say that after tinkering with the JAVA API and then looking at
straight REST calls (or use GitHub - searchbox-io/Jest: Elasticsearch Java Rest Client.), I'm
leaning away from the JAVA API. I'm not sure it buys me anything to use
that.

It still scares me to have client<->server architecture for management apps
with no business tier in between. Maybe I'm just old. :wink:

On Thursday, May 2, 2013 11:14:53 AM UTC-4, Roy Russo wrote:

Hello All,

I have recently embarked on building a Management and Monitoring app for
Elasticsearch. I do have a few questions on what is most feasible, however:

  1. I have decided to use the JAVA API. It is what ES uses internally, so
    it makes the most sense from a long-term perspective.

  2. Not to be used as a plugin: For monitoring clustered configs in a
    real-world scenario (think 'enterprise), I don't see mass adoption of a
    plugin, but a downloadable war would be most likely to be used.

  3. I could not find anywhere if ES has capabilities to persist usage
    information. Something I would like to add to the app is a way to view
    Analytics and usage information.

Feel free to share your thoughts and feedback on my approach.

Regards,
Roy Russo
http://www.linkedin.com/in/royrusso

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

7

Hi,

just a comment (or idea if you will). If you want to build a system that
would pull data from elasticsearch cluster (for later analysis) I would
definitely consider pulling from more (if not from ALL) available nodes
from the cluster (assuming size of stored data is not a problem). The
reason being that if you pull only from a single node then you can face two
critical situations:

  • that node fails and you have no data since that point in time
  • cluster suffer from split brain and then you see only the info that comes
    from the part your node is part of. I think that especially in case of
    split brain it can be very useful to see "whole picture". And you can
    cleanup or drop duplicit data later anyway if you need.

Not sure if SPM does this (if yes, then big +1 and kudos to Sematext).

Another possible inspiration can be ES river that can pull data from
cluster and store it again into (not necessarily the same) cluster. Like

Regards,
Lukas

On Thu, May 2, 2013 at 9:28 PM, Roy Russo royrusso@gmail.com wrote:

Hi Otis,

1/ It looks like the TransportClient will return all of the data bits that
are exposed via REST - including jvm and system metrics, so there is no
need to drop an agent on the server. I'm testing this locally now and seems
to work.

2/ I looked at SPM just now. I love the ease of registration - nice job.
The only difference I would say here is that there is no need to deploy
anything on the server.

This system would be much like Ganglia - hopefully easier on the eyes. The
one downside I see is that is the development team makes
reverse-incompatible changes, my code will fail miserably, as I bundle the
LATEST in my app, but I guess the same would go for any changes in the REST
responses.

On Thursday, May 2, 2013 3:04:53 PM UTC-4, Otis Gospodnetic wrote:

Hi,

Not sure I follow.
Sure, you can run a remote agent and pull metrics out of ES. But you may
not be able to get to server metrics, for example. So you need a local
agent. Re too much data - up to the agent what it gets, how often, whether
it compresses, etc.

I won't speak for ES head and BigDesk, but what you are describing
doesn't sound any different than SPM for example. Or how people set up
Ganglia. Or ... but maybe I'm missing something? :slight_smile:

Otis

Search Analytics - http://sematext.com/search-**analytics/index.htmlhttp://sematext.com/search-analytics/index.html
ELASTICSEARCH Performance Monitoring - Sematext Monitoring | Infrastructure Monitoring Service**
index.html http://sematext.com/spm/index.html

On Thursday, May 2, 2013 2:25:45 PM UTC-4, Roy Russo wrote:

Thanks Otis. I did see those, but I'm looking to build something a bit
more full-featured that can one day use persistence for historic analysis,
making changes/remembering the cluster config, etc...

Other reasons:

  1. Most common (enterprise?) infrastructure and cloud monitoring apps
    are not normally on the actual running instance of the application one
    wants to monitor/manage. Sometimes those machines are not accessible via
    port 80 and it also risks adding overhead on the instance being monitored.
    So a hub-spoke topography for monitoring is what I prefer for scalability
    reasons.
  2. This is a theoretical, but I feel it is a heavier burden to offload
    all of the logic and data to the client. Instead of atomic calls that can
    be filtered, you risk ending up with "too much" data in the client memory
    and flowing through the pipes.
  3. It is easier to add the capability for a server-side app to be used
    in a mutli-tenant model, ie. offering monitoring/management solutions to ES
    users.

Thoughts?

On Thursday, May 2, 2013 11:14:53 AM UTC-4, Roy Russo wrote:

Hello All,

I have recently embarked on building a Management and Monitoring app
for Elasticsearch. I do have a few questions on what is most feasible,
however:

  1. I have decided to use the JAVA API. It is what ES uses internally,
    so it makes the most sense from a long-term perspective.

  2. Not to be used as a plugin: For monitoring clustered configs in a
    real-world scenario (think 'enterprise), I don't see mass adoption of a
    plugin, but a downloadable war would be most likely to be used.

  3. I could not find anywhere if ES has capabilities to persist usage
    information. Something I would like to add to the app is a way to view
    Analytics and usage information.

Feel free to share your thoughts and feedback on my approach.

Regards,
Roy Russo
http://www.linkedin.com/in/**royrussohttp://www.linkedin.com/in/royrusso

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

8

Good point, Lukas. (Good to see another JBoss dev here. Well, I was one.
:wink:

I referenced this post when I started:
https://groups.google.com/forum/?fromgroups=#!searchin/elasticsearch/rest$20api/elasticsearch/TihFMAJjTR4/7tSMdtGlHNIJ

And I think he came to the same conclusion as I. After looking at the
Searchbox.io JEST API, it is a bit crippled, in that monitoring and
management calls are inexplicably missing, so I'm back to the ES REST API -
in favor of the TransportClient.

On ES river-info - that is something I was looking for - using ES to store
it's own metric data, and being able then to display it on the monitoring
app. Thanks!

On Thursday, May 2, 2013 11:14:53 AM UTC-4, Roy Russo wrote:

Hello All,

I have recently embarked on building a Management and Monitoring app for
Elasticsearch. I do have a few questions on what is most feasible, however:

  1. I have decided to use the JAVA API. It is what ES uses internally, so
    it makes the most sense from a long-term perspective.

  2. Not to be used as a plugin: For monitoring clustered configs in a
    real-world scenario (think 'enterprise), I don't see mass adoption of a
    plugin, but a downloadable war would be most likely to be used.

  3. I could not find anywhere if ES has capabilities to persist usage
    information. Something I would like to add to the app is a way to view
    Analytics and usage information.

Feel free to share your thoughts and feedback on my approach.

Regards,
Roy Russo
http://www.linkedin.com/in/royrusso

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

9

Yes, SPM does not suffer from split brain or ES node failure situations.
We did explicitly take that into consideration when deciding how to design
SPM agent for ES.

So I'll happily take that +1 and kudos Lukas and take it with the SPM dev
team here at Sematext, thanks! :slight_smile:

Otis

Search Analytics - Cloud Monitoring Tools & Services | Sematext
ELASTICSEARCH Performance Monitoring - Sematext Monitoring | Infrastructure Monitoring Service

On Thursday, May 2, 2013 3:41:29 PM UTC-4, Lukáš Vlček wrote:

Hi,

just a comment (or idea if you will). If you want to build a system that
would pull data from elasticsearch cluster (for later analysis) I would
definitely consider pulling from more (if not from ALL) available nodes
from the cluster (assuming size of stored data is not a problem). The
reason being that if you pull only from a single node then you can face two
critical situations:

  • that node fails and you have no data since that point in time
  • cluster suffer from split brain and then you see only the info that
    comes from the part your node is part of. I think that especially in case
    of split brain it can be very useful to see "whole picture". And you can
    cleanup or drop duplicit data later anyway if you need.

Not sure if SPM does this (if yes, then big +1 and kudos to Sematext).

Another possible inspiration can be ES river that can pull data from
cluster and store it again into (not necessarily the same) cluster. Like
GitHub - searchisko/elasticsearch-river-sysinfo: River component for elasticsearch which collects system informations from ES cluster in defined intervals and stores them into search indices, so can be analyzed later.

Regards,
Lukas

On Thu, May 2, 2013 at 9:28 PM, Roy Russo <royr...@gmail.com <javascript:>

wrote:

Hi Otis,

1/ It looks like the TransportClient will return all of the data bits
that are exposed via REST - including jvm and system metrics, so there is
no need to drop an agent on the server. I'm testing this locally now and
seems to work.

2/ I looked at SPM just now. I love the ease of registration - nice job.
The only difference I would say here is that there is no need to deploy
anything on the server.

This system would be much like Ganglia - hopefully easier on the eyes.
The one downside I see is that is the development team makes
reverse-incompatible changes, my code will fail miserably, as I bundle the
LATEST in my app, but I guess the same would go for any changes in the REST
responses.

On Thursday, May 2, 2013 3:04:53 PM UTC-4, Otis Gospodnetic wrote:

Hi,

Not sure I follow.
Sure, you can run a remote agent and pull metrics out of ES. But you
may not be able to get to server metrics, for example. So you need a local
agent. Re too much data - up to the agent what it gets, how often, whether
it compresses, etc.

I won't speak for ES head and BigDesk, but what you are describing
doesn't sound any different than SPM for example. Or how people set up
Ganglia. Or ... but maybe I'm missing something? :slight_smile:

Otis

Search Analytics - http://sematext.com/search-**analytics/index.htmlhttp://sematext.com/search-analytics/index.html
ELASTICSEARCH Performance Monitoring - Sematext Monitoring | Infrastructure Monitoring Service**
index.html http://sematext.com/spm/index.html

On Thursday, May 2, 2013 2:25:45 PM UTC-4, Roy Russo wrote:

Thanks Otis. I did see those, but I'm looking to build something a bit
more full-featured that can one day use persistence for historic analysis,
making changes/remembering the cluster config, etc...

Other reasons:

  1. Most common (enterprise?) infrastructure and cloud monitoring apps
    are not normally on the actual running instance of the application one
    wants to monitor/manage. Sometimes those machines are not accessible via
    port 80 and it also risks adding overhead on the instance being monitored.
    So a hub-spoke topography for monitoring is what I prefer for scalability
    reasons.
  2. This is a theoretical, but I feel it is a heavier burden to offload
    all of the logic and data to the client. Instead of atomic calls that can
    be filtered, you risk ending up with "too much" data in the client memory
    and flowing through the pipes.
  3. It is easier to add the capability for a server-side app to be used
    in a mutli-tenant model, ie. offering monitoring/management solutions to ES
    users.

Thoughts?

On Thursday, May 2, 2013 11:14:53 AM UTC-4, Roy Russo wrote:

Hello All,

I have recently embarked on building a Management and Monitoring app
for Elasticsearch. I do have a few questions on what is most feasible,
however:

  1. I have decided to use the JAVA API. It is what ES uses internally,
    so it makes the most sense from a long-term perspective.

  2. Not to be used as a plugin: For monitoring clustered configs in a
    real-world scenario (think 'enterprise), I don't see mass adoption of a
    plugin, but a downloadable war would be most likely to be used.

  3. I could not find anywhere if ES has capabilities to persist usage
    information. Something I would like to add to the app is a way to view
    Analytics and usage information.

Feel free to share your thoughts and feedback on my approach.

Regards,
Roy Russo
http://www.linkedin.com/in/**royrussohttp://www.linkedin.com/in/royrusso

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

10

Yea Otis, good move! :slight_smile:

On the other hand, when you take the river approach - river is a singleton
(per cluster) so if the split brain happens, then each half of the cluster
should have its own river started, no? :slight_smile: Haven't tested this myself but
should work if river configured appropriately (i.e. the river is allowed on
most of the nodes). So with river this could theoretically work as well.

Lukas

On Thu, May 2, 2013 at 11:11 PM, Otis Gospodnetic <
otis.gospodnetic@gmail.com> wrote:

Yes, SPM does not suffer from split brain or ES node failure situations.
We did explicitly take that into consideration when deciding how to design
SPM agent for ES.

So I'll happily take that +1 and kudos Lukas and take it with the SPM dev
team here at Sematext, thanks! :slight_smile:

Otis

Search Analytics - http://sematext.com/search-**analytics/index.htmlhttp://sematext.com/search-analytics/index.html
ELASTICSEARCH Performance Monitoring - Sematext Monitoring | Infrastructure Monitoring Service**
index.html http://sematext.com/spm/index.html

On Thursday, May 2, 2013 3:41:29 PM UTC-4, Lukáš Vlček wrote:

Hi,

just a comment (or idea if you will). If you want to build a system that
would pull data from elasticsearch cluster (for later analysis) I would
definitely consider pulling from more (if not from ALL) available nodes
from the cluster (assuming size of stored data is not a problem). The
reason being that if you pull only from a single node then you can face two
critical situations:

  • that node fails and you have no data since that point in time
  • cluster suffer from split brain and then you see only the info that
    comes from the part your node is part of. I think that especially in case
    of split brain it can be very useful to see "whole picture". And you can
    cleanup or drop duplicit data later anyway if you need.

Not sure if SPM does this (if yes, then big +1 and kudos to Sematext).

Another possible inspiration can be ES river that can pull data from
cluster and store it again into (not necessarily the same) cluster. Like
https://github.com/**jbossorg/elasticsearch-river-**sysinfohttps://github.com/jbossorg/elasticsearch-river-sysinfo

Regards,
Lukas

On Thu, May 2, 2013 at 9:28 PM, Roy Russo royr...@gmail.com wrote:

Hi Otis,

1/ It looks like the TransportClient will return all of the data bits
that are exposed via REST - including jvm and system metrics, so there is
no need to drop an agent on the server. I'm testing this locally now and
seems to work.

2/ I looked at SPM just now. I love the ease of registration - nice job.
The only difference I would say here is that there is no need to deploy
anything on the server.

This system would be much like Ganglia - hopefully easier on the eyes.
The one downside I see is that is the development team makes
reverse-incompatible changes, my code will fail miserably, as I bundle the
LATEST in my app, but I guess the same would go for any changes in the REST
responses.

On Thursday, May 2, 2013 3:04:53 PM UTC-4, Otis Gospodnetic wrote:

Hi,

Not sure I follow.
Sure, you can run a remote agent and pull metrics out of ES. But you
may not be able to get to server metrics, for example. So you need a local
agent. Re too much data - up to the agent what it gets, how often, whether
it compresses, etc.

I won't speak for ES head and BigDesk, but what you are describing
doesn't sound any different than SPM for example. Or how people set up
Ganglia. Or ... but maybe I'm missing something? :slight_smile:

Otis

Search Analytics - http://sematext.com/search-**a**nalytics/index.htmlhttp://sematext.com/search-analytics/index.html
ELASTICSEARCH Performance Monitoring - http://sematext.com/spm/**inde**
x.html http://sematext.com/spm/index.html

On Thursday, May 2, 2013 2:25:45 PM UTC-4, Roy Russo wrote:

Thanks Otis. I did see those, but I'm looking to build something a bit
more full-featured that can one day use persistence for historic analysis,
making changes/remembering the cluster config, etc...

Other reasons:

  1. Most common (enterprise?) infrastructure and cloud monitoring apps
    are not normally on the actual running instance of the application one
    wants to monitor/manage. Sometimes those machines are not accessible via
    port 80 and it also risks adding overhead on the instance being monitored.
    So a hub-spoke topography for monitoring is what I prefer for scalability
    reasons.
  2. This is a theoretical, but I feel it is a heavier burden to offload
    all of the logic and data to the client. Instead of atomic calls that can
    be filtered, you risk ending up with "too much" data in the client memory
    and flowing through the pipes.
  3. It is easier to add the capability for a server-side app to be used
    in a mutli-tenant model, ie. offering monitoring/management solutions to ES
    users.

Thoughts?

On Thursday, May 2, 2013 11:14:53 AM UTC-4, Roy Russo wrote:

Hello All,

I have recently embarked on building a Management and Monitoring app
for Elasticsearch. I do have a few questions on what is most feasible,
however:

  1. I have decided to use the JAVA API. It is what ES uses internally,
    so it makes the most sense from a long-term perspective.

  2. Not to be used as a plugin: For monitoring clustered configs in a
    real-world scenario (think 'enterprise), I don't see mass adoption of a
    plugin, but a downloadable war would be most likely to be used.

  3. I could not find anywhere if ES has capabilities to persist usage
    information. Something I would like to add to the app is a way to view
    Analytics and usage information.

Feel free to share your thoughts and feedback on my approach.

Regards,
Roy Russo
http://www.linkedin.com/in/**roy**russohttp://www.linkedin.com/in/royrusso

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@**googlegroups.com.

For more options, visit https://groups.google.com/**groups/opt_outhttps://groups.google.com/groups/opt_out
.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

11

On Thu, May 2, 2013 at 10:02 PM, Roy Russo royrusso@gmail.com wrote:

Good point, Lukas. (Good to see another JBoss dev here. Well, I was one.
:wink:

Welcome :slight_smile:

On ES river-info - that is something I was looking for - using ES to store
it's own metric data, and being able then to display it on the monitoring
app. Thanks!

Just note, this river is pretty much still concept. We do not use this is
production now. Also I think what needs to be done (except GUI and other
fancy logic) is to allow to store other custom defined metrics alongside
those stored by the river (for example metric about custom queries and
searches or other metrics from related systems) so that more interesting
analysis would be possible.

Also one other thing I was thinking about is what kind of metrics
calculations one could get out of all these stored data in Elasticsearch. I
mean analytics calculated directly by Elasticsearch. As of now I am still
not sure if Elasticsearch can really give you all important aggregations
out of the box. For example I am not sure if it can give you percentiles
(median and 95th percentile in this case) although after today 0.90.0
introduction webinar I think it might be possible in the future (given
planned faceting improvements).

So when SPM decided to store the data in HBase and not in Elasticsearch,
this might be better option for now? Maybe... just take this as a random
thoughts of someone who desperately needs more sleep.

Lukas

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.