Visibility


(smonasco-2) #1

Hi,

I'm trying to get a lot more visibility and metrics into what's going on
under the hood.

Occasionally, we see spikes in memory. I'd like to get heap mem used on a
per shard basis. If I'm not mistaken, somewhere somehow, this Lucene index
that is a shard is using memory in the heap, and I'd like to collect metric.

It may also be an operation somewhere higher up in the elasticsearch level
where we are merging results from shards or results from indexes (maybe
elasticsearch doesn't bother to merge twice but merges once), that's also a
mem space I'd like to collect data on.

I think a per query mem use would also be something interesting, though,
perhaps obviously too much to keep up with for every query (maybe a future
opt-in feature, unless it's already there and I'm missing it).

Other cluster events like nodes entering and exiting the cluster or the
changing of the master would be nice to collect.

I'm guessing some of this isn't available and some of it is, but my
Google-Fu seems to be lacking. I'm pretty sure I can poll to figure out
the events happened, but was wondering if there was something in the java
client node where I could get a Future or some other hook to turn it into a
push instead of a pull.

Any help will be appreciated. I'm aware it's a wide net though.

--Shannon Monasco

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/56362f94-c20b-4201-ae15-5f5f9ca77ff4%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(Mark Walkom) #2

Depends what you want to do really.

There are plugins like ElasticHQ, Marvel, kopf and bigdesk that will give
you some info. You can also hook collectd into the stack and take metrics,
or use plugins from nagios etc.
What monitoring platforms do you have in place now?

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: markw@campaignmonitor.com
web: www.campaignmonitor.com

On 3 July 2014 07:49, smonasco smonasco@gmail.com wrote:

Hi,

I'm trying to get a lot more visibility and metrics into what's going on
under the hood.

Occasionally, we see spikes in memory. I'd like to get heap mem used on a
per shard basis. If I'm not mistaken, somewhere somehow, this Lucene index
that is a shard is using memory in the heap, and I'd like to collect metric.

It may also be an operation somewhere higher up in the elasticsearch level
where we are merging results from shards or results from indexes (maybe
elasticsearch doesn't bother to merge twice but merges once), that's also a
mem space I'd like to collect data on.

I think a per query mem use would also be something interesting, though,
perhaps obviously too much to keep up with for every query (maybe a future
opt-in feature, unless it's already there and I'm missing it).

Other cluster events like nodes entering and exiting the cluster or the
changing of the master would be nice to collect.

I'm guessing some of this isn't available and some of it is, but my
Google-Fu seems to be lacking. I'm pretty sure I can poll to figure out
the events happened, but was wondering if there was something in the java
client node where I could get a Future or some other hook to turn it into a
push instead of a pull.

Any help will be appreciated. I'm aware it's a wide net though.

--Shannon Monasco

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/56362f94-c20b-4201-ae15-5f5f9ca77ff4%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/56362f94-c20b-4201-ae15-5f5f9ca77ff4%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEM624aCuHQhWVHp0MOrTZH3s0y6kN7jqkg7bXEQF%2BrtwfEqTQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


(smonasco-2) #3

I currently record basically everything in bigdesk, all the numerics from
cluster health, cluster state, nodes info, node stats, index status and
segments.

I want memory allocated on a per shard level for Lucene level actions,
query level actions (outside field and filter cache) and hooks into events
like nodes entering and exiting the cluster, new indexes, alias and other
administrative changes and master elections.

Basically when it comes to memory I'd like to have all parts of the heap
accounted for. Field + filter cache is not accounting for whatever process
is spiking nor does it answer most of the heap. At 29 gigs being used and
garbage collection taking minutes, but not getting anything, elastic is
only reporting 7 gigs in cache. We can discuss my particular memory
problems and solutions, but mostly I'm after the visibility.

--Shannon Monasco
On Jul 2, 2014 5:50 PM, "Mark Walkom" markw@campaignmonitor.com wrote:

Depends what you want to do really.

There are plugins like ElasticHQ, Marvel, kopf and bigdesk that will give
you some info. You can also hook collectd into the stack and take metrics,
or use plugins from nagios etc.
What monitoring platforms do you have in place now?

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: markw@campaignmonitor.com
web: www.campaignmonitor.com

On 3 July 2014 07:49, smonasco smonasco@gmail.com wrote:

Hi,

I'm trying to get a lot more visibility and metrics into what's going on
under the hood.

Occasionally, we see spikes in memory. I'd like to get heap mem used on
a per shard basis. If I'm not mistaken, somewhere somehow, this Lucene
index that is a shard is using memory in the heap, and I'd like to collect
metric.

It may also be an operation somewhere higher up in the elasticsearch
level where we are merging results from shards or results from indexes
(maybe elasticsearch doesn't bother to merge twice but merges once), that's
also a mem space I'd like to collect data on.

I think a per query mem use would also be something interesting, though,
perhaps obviously too much to keep up with for every query (maybe a future
opt-in feature, unless it's already there and I'm missing it).

Other cluster events like nodes entering and exiting the cluster or the
changing of the master would be nice to collect.

I'm guessing some of this isn't available and some of it is, but my
Google-Fu seems to be lacking. I'm pretty sure I can poll to figure out
the events happened, but was wondering if there was something in the java
client node where I could get a Future or some other hook to turn it into a
push instead of a pull.

Any help will be appreciated. I'm aware it's a wide net though.

--Shannon Monasco

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/56362f94-c20b-4201-ae15-5f5f9ca77ff4%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/56362f94-c20b-4201-ae15-5f5f9ca77ff4%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/sF_C58d96ms/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAEM624aCuHQhWVHp0MOrTZH3s0y6kN7jqkg7bXEQF%2BrtwfEqTQ%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAEM624aCuHQhWVHp0MOrTZH3s0y6kN7jqkg7bXEQF%2BrtwfEqTQ%40mail.gmail.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAFDU5WJYc8LbWyqY%2Bu8u%2BcS_WbgNBOfzNEKOLT_xV4pwnfQj_Q%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


(Michael Hart) #4

I strongly recommend Marvel (and I don't work for elasticsearch), it's
quite detailed and you get can insight into exactly what elasticsearch is
doing. The only thing it doesn't have full visibility into is the detailed
GC stats, for those you'll have to enable GC logging and use a gcviewer to
investigate.

I also have collectd running with the python module enabled and then this
plugin: https://github.com/phobos182/collectd-elasticsearch but that's only
to tie it into our alerting system.

mike

On Wednesday, July 2, 2014 10:33:46 PM UTC-4, smonasco wrote:

I currently record basically everything in bigdesk, all the numerics from
cluster health, cluster state, nodes info, node stats, index status and
segments.

I want memory allocated on a per shard level for Lucene level actions,
query level actions (outside field and filter cache) and hooks into events
like nodes entering and exiting the cluster, new indexes, alias and other
administrative changes and master elections.

Basically when it comes to memory I'd like to have all parts of the heap
accounted for. Field + filter cache is not accounting for whatever process
is spiking nor does it answer most of the heap. At 29 gigs being used and
garbage collection taking minutes, but not getting anything, elastic is
only reporting 7 gigs in cache. We can discuss my particular memory
problems and solutions, but mostly I'm after the visibility.

--Shannon Monasco
On Jul 2, 2014 5:50 PM, "Mark Walkom" <ma...@campaignmonitor.com
<javascript:>> wrote:

Depends what you want to do really.

There are plugins like ElasticHQ, Marvel, kopf and bigdesk that will give
you some info. You can also hook collectd into the stack and take metrics,
or use plugins from nagios etc.
What monitoring platforms do you have in place now?

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: ma...@campaignmonitor.com <javascript:>
web: www.campaignmonitor.com

On 3 July 2014 07:49, smonasco <smon...@gmail.com <javascript:>> wrote:

Hi,

I'm trying to get a lot more visibility and metrics into what's going on
under the hood.

Occasionally, we see spikes in memory. I'd like to get heap mem used on
a per shard basis. If I'm not mistaken, somewhere somehow, this Lucene
index that is a shard is using memory in the heap, and I'd like to collect
metric.

It may also be an operation somewhere higher up in the elasticsearch
level where we are merging results from shards or results from indexes
(maybe elasticsearch doesn't bother to merge twice but merges once), that's
also a mem space I'd like to collect data on.

I think a per query mem use would also be something interesting, though,
perhaps obviously too much to keep up with for every query (maybe a future
opt-in feature, unless it's already there and I'm missing it).

Other cluster events like nodes entering and exiting the cluster or the
changing of the master would be nice to collect.

I'm guessing some of this isn't available and some of it is, but my
Google-Fu seems to be lacking. I'm pretty sure I can poll to figure out
the events happened, but was wondering if there was something in the java
client node where I could get a Future or some other hook to turn it into a
push instead of a pull.

Any help will be appreciated. I'm aware it's a wide net though.

--Shannon Monasco

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/56362f94-c20b-4201-ae15-5f5f9ca77ff4%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/56362f94-c20b-4201-ae15-5f5f9ca77ff4%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/sF_C58d96ms/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAEM624aCuHQhWVHp0MOrTZH3s0y6kN7jqkg7bXEQF%2BrtwfEqTQ%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAEM624aCuHQhWVHp0MOrTZH3s0y6kN7jqkg7bXEQF%2BrtwfEqTQ%40mail.gmail.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/3246f705-d3ac-4b8f-beb8-2955dd2c9d06%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(smonasco-2) #5

Thanks. I'll have to check that out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/6fb48f2b-d690-40bd-a01c-649849ee1388%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(smonasco-2) #6

Thanks. I'll have to check that out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/51693680-2a0b-4ac4-a3b5-e0c721773701%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(system) #7