On Wed, Aug 17, 2011 at 4:00 PM, Paul Smith tallpsmith@gmail.com wrote:
On 17 August 2011 21:49, Shay Banon kimchy@gmail.com wrote:
Heya,
I think that something like this should be built as a feature in
elasticsearch. Stats collection is something important, and we should have a
module for that, that both reports the stats and also allow for custom stats
"publishing" options. Lets work on getting this into the "core" of
elasticsearch?
On the one hand I agree, on the other, there's a million ways to do this.
We have OpenTSDB with it's collector service, BigDesk etc. Lots of people
will have different use cases, and the data being exported by the REST api
is pretty darn useful as a general mechanism.
Right. The idea that I had is there will be internal structures
that accumulate "stats" or provide them as an API. There will also be a way
to register for stats events internally, though I am not sure you need it.
Once we have it as an API, then the plugin can periodically call those stats
API internally, and report them to whatever monitoring system one so
chooses.
For example, if we have am "index stats" module, that keeps count of some
counters (like number of invocations, total time), then the custom monitor
plugin can call this periodically, and report those stats to an externa
system.
My embryonic plugin is located here:
https://github.com/Aconex/elasticsearch-pcp
The docs there should work for 0.17.6.
it's not exporting much just yet, and I don't have enough info in the
README file as a mini tutorial yet to show the true power you can do with
PCP. Because the interception bit is 'missing' I can't really show the
really cool bit just yet. If we can 'find a way' to allow this interception
back, then I'll be able to bring that functionality back and demo it in the
plugin form. We can continue to discuss after that about what elasticsearch
should have in it or not.
Regarding the netty aspect, it should be simple to add, but again with
some internal work. Open an issue for it. That one is a simple one.
I've managed to find the NetworkStats object being exposed in the
NetworkService and can use that for export. that's useful, but not the same
as the bandwidth level info I think would be useful.
I'll open an issue. Thanks Shay!
Paul