I'm trying to get a lot more visibility and metrics into what's going on
under the hood.
Occasionally, we see spikes in memory. I'd like to get heap mem used on a
per shard basis. If I'm not mistaken, somewhere somehow, this Lucene index
that is a shard is using memory in the heap, and I'd like to collect metric.
It may also be an operation somewhere higher up in the elasticsearch level
where we are merging results from shards or results from indexes (maybe
elasticsearch doesn't bother to merge twice but merges once), that's also a
mem space I'd like to collect data on.
I think a per query mem use would also be something interesting, though,
perhaps obviously too much to keep up with for every query (maybe a future
opt-in feature, unless it's already there and I'm missing it).
Other cluster events like nodes entering and exiting the cluster or the
changing of the master would be nice to collect.
I'm guessing some of this isn't available and some of it is, but my
Google-Fu seems to be lacking. I'm pretty sure I can poll to figure out
the events happened, but was wondering if there was something in the java
client node where I could get a Future or some other hook to turn it into a
push instead of a pull.
Any help will be appreciated. I'm aware it's a wide net though.
There are plugins like ElasticHQ, Marvel, kopf and bigdesk that will give
you some info. You can also hook collectd into the stack and take metrics,
or use plugins from nagios etc.
What monitoring platforms do you have in place now?
I'm trying to get a lot more visibility and metrics into what's going on
under the hood.
Occasionally, we see spikes in memory. I'd like to get heap mem used on a
per shard basis. If I'm not mistaken, somewhere somehow, this Lucene index
that is a shard is using memory in the heap, and I'd like to collect metric.
It may also be an operation somewhere higher up in the elasticsearch level
where we are merging results from shards or results from indexes (maybe
elasticsearch doesn't bother to merge twice but merges once), that's also a
mem space I'd like to collect data on.
I think a per query mem use would also be something interesting, though,
perhaps obviously too much to keep up with for every query (maybe a future
opt-in feature, unless it's already there and I'm missing it).
Other cluster events like nodes entering and exiting the cluster or the
changing of the master would be nice to collect.
I'm guessing some of this isn't available and some of it is, but my
Google-Fu seems to be lacking. I'm pretty sure I can poll to figure out
the events happened, but was wondering if there was something in the java
client node where I could get a Future or some other hook to turn it into a
push instead of a pull.
Any help will be appreciated. I'm aware it's a wide net though.
I currently record basically everything in bigdesk, all the numerics from
cluster health, cluster state, nodes info, node stats, index status and
segments.
I want memory allocated on a per shard level for Lucene level actions,
query level actions (outside field and filter cache) and hooks into events
like nodes entering and exiting the cluster, new indexes, alias and other
administrative changes and master elections.
Basically when it comes to memory I'd like to have all parts of the heap
accounted for. Field + filter cache is not accounting for whatever process
is spiking nor does it answer most of the heap. At 29 gigs being used and
garbage collection taking minutes, but not getting anything, elastic is
only reporting 7 gigs in cache. We can discuss my particular memory
problems and solutions, but mostly I'm after the visibility.
There are plugins like ElasticHQ, Marvel, kopf and bigdesk that will give
you some info. You can also hook collectd into the stack and take metrics,
or use plugins from nagios etc.
What monitoring platforms do you have in place now?
I'm trying to get a lot more visibility and metrics into what's going on
under the hood.
Occasionally, we see spikes in memory. I'd like to get heap mem used on
a per shard basis. If I'm not mistaken, somewhere somehow, this Lucene
index that is a shard is using memory in the heap, and I'd like to collect
metric.
It may also be an operation somewhere higher up in the elasticsearch
level where we are merging results from shards or results from indexes
(maybe elasticsearch doesn't bother to merge twice but merges once), that's
also a mem space I'd like to collect data on.
I think a per query mem use would also be something interesting, though,
perhaps obviously too much to keep up with for every query (maybe a future
opt-in feature, unless it's already there and I'm missing it).
Other cluster events like nodes entering and exiting the cluster or the
changing of the master would be nice to collect.
I'm guessing some of this isn't available and some of it is, but my
Google-Fu seems to be lacking. I'm pretty sure I can poll to figure out
the events happened, but was wondering if there was something in the java
client node where I could get a Future or some other hook to turn it into a
push instead of a pull.
Any help will be appreciated. I'm aware it's a wide net though.
I strongly recommend Marvel (and I don't work for elasticsearch), it's
quite detailed and you get can insight into exactly what elasticsearch is
doing. The only thing it doesn't have full visibility into is the detailed
GC stats, for those you'll have to enable GC logging and use a gcviewer to
investigate.
On Wednesday, July 2, 2014 10:33:46 PM UTC-4, smonasco wrote:
I currently record basically everything in bigdesk, all the numerics from
cluster health, cluster state, nodes info, node stats, index status and
segments.
I want memory allocated on a per shard level for Lucene level actions,
query level actions (outside field and filter cache) and hooks into events
like nodes entering and exiting the cluster, new indexes, alias and other
administrative changes and master elections.
Basically when it comes to memory I'd like to have all parts of the heap
accounted for. Field + filter cache is not accounting for whatever process
is spiking nor does it answer most of the heap. At 29 gigs being used and
garbage collection taking minutes, but not getting anything, elastic is
only reporting 7 gigs in cache. We can discuss my particular memory
problems and solutions, but mostly I'm after the visibility.
There are plugins like ElasticHQ, Marvel, kopf and bigdesk that will give
you some info. You can also hook collectd into the stack and take metrics,
or use plugins from nagios etc.
What monitoring platforms do you have in place now?
On 3 July 2014 07:49, smonasco <smon...@gmail.com <javascript:>> wrote:
Hi,
I'm trying to get a lot more visibility and metrics into what's going on
under the hood.
Occasionally, we see spikes in memory. I'd like to get heap mem used on
a per shard basis. If I'm not mistaken, somewhere somehow, this Lucene
index that is a shard is using memory in the heap, and I'd like to collect
metric.
It may also be an operation somewhere higher up in the elasticsearch
level where we are merging results from shards or results from indexes
(maybe elasticsearch doesn't bother to merge twice but merges once), that's
also a mem space I'd like to collect data on.
I think a per query mem use would also be something interesting, though,
perhaps obviously too much to keep up with for every query (maybe a future
opt-in feature, unless it's already there and I'm missing it).
Other cluster events like nodes entering and exiting the cluster or the
changing of the master would be nice to collect.
I'm guessing some of this isn't available and some of it is, but my
Google-Fu seems to be lacking. I'm pretty sure I can poll to figure out
the events happened, but was wondering if there was something in the java
client node where I could get a Future or some other hook to turn it into a
push instead of a pull.
Any help will be appreciated. I'm aware it's a wide net though.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.