Guice version/plugin & bindInterceptor method.. .?

Paul_Smith · August 17, 2011, 2:27am

I had an elasticsearch fork which I'm converting to a plugin (sooo much
nicer way of doing it). However the code I had in my custom module extended
from AbstractModule and used an inherited method:

protected void

bindInterceptor(org.elasticsearch.common.inject.matcher.Matcher<? super
java.lang.Class<?>> classMatcher,
org.elasticsearch.common.inject.matcher.Matcher<? super
java.lang.reflect.Method> methodMatcher,
org.elasticsearch.common.aopalliance.intercept.MethodInterceptor...
interceptors) { /* compiled code */ }

which WAS on the AbstractModule class a while ago, but as of trying to get
this working under 0.17.6 is no longer there, which I suspect is a
difference in Google guice which has been shaded into the
org.elascticsearch.common.inject* area.

The Google Guice code examples still refer to this availability though, so
I'm just not sure what I need to do.. ?

any tips?

tah,

Paul Smith

kimchy · August 17, 2011, 3:27am

elasticsearch has embedded guice 2.0 codebase since it uses a narrow set of
features form guice, and 3.0 is considerably slower than 2.0. As part of the
embedding process, a version guice without AOP was embedded in order to try
and reduce the size of the jar, as well as not to use cglib and the like
for enhancements (they are faster to create, but an overhead loaded classes
wise).

Are you using AOP? If so, why? Is there a chance to simply move that logic
to a specific callback based design (which will be faster)?

On Wed, Aug 17, 2011 at 5:27 AM, Paul Smith tallpsmith@gmail.com wrote:

I had an elasticsearch fork which I'm converting to a plugin (sooo much
nicer way of doing it). However the code I had in my custom module extended
from AbstractModule and used an inherited method:
protected void
bindInterceptor(org.elasticsearch.common.inject.matcher.Matcher<? super java.lang.Class<?>> classMatcher,
org.elasticsearch.common.inject.matcher.Matcher<? super
java.lang.reflect.Method> methodMatcher,
org.elasticsearch.common.aopalliance.intercept.MethodInterceptor...
interceptors) { /* compiled code */ }

which WAS on the AbstractModule class a while ago, but as of trying to get
this working under 0.17.6 is no longer there, which I suspect is a
difference in Google guice which has been shaded into the
org.elascticsearch.common.inject* area.

The Google Guice code examples still refer to this availability though, so
I'm just not sure what I need to do.. ?

GitHub - google/guice: Guice (pronounced 'juice') is a lightweight dependency injection framework for Java 8 and above, brought to you by Google.

any tips?

tah,

Paul Smith

Paul_Smith · August 17, 2011, 5:12am

On 17 August 2011 13:27, Shay Banon kimchy@gmail.com wrote:

elasticsearch has embedded guice 2.0 codebase since it uses a narrow set of
features form guice, and 3.0 is considerably slower than 2.0. As part of the
embedding process, a version guice without AOP was embedded in order to try
and reduce the size of the jar, as well as not to use cglib and the like
for enhancements (they are faster to create, but an overhead loaded classes
wise).

Are you using AOP? If so, why? Is there a chance to simply move that logic
to a specific callback based design (which will be faster)?

Originally, I had something like this:

gist.github.com

https://gist.github.com/tallpsmith/1150846

gistfile1.java

            bindEngineMethodToCounterAndAction(parfaitService, RobinEngine.class.getMethod("create", Engine.Create.class));
            bindEngineMethodToCounterAndAction(parfaitService, RobinEngine.class.getMethod("index", Engine.Index.class));
            bindEngineMethodToCounterAndAction(parfaitService, RobinEngine.class.getMethod("flush", Engine.Flush.class));
            bindEngineMethodToCounterAndAction(parfaitService, RobinEngine.class.getMethod("delete", Engine.Delete.class));
            bindEngineMethodToCounterAndAction(parfaitService, RobinEngine.class.getMethod("delete", Engine.DeleteByQuery.class));
            bindEngineMethodToCounterAndAction(parfaitService, RobinEngine.class.getMethod("optimize", Engine.Optimize.class));
            bindEngineMethodToCounterAndAction(parfaitService, RobinEngine.class.getMethod("refresh", Engine.Refresh.class));
            bindEngineMethodToCounterAndAction(parfaitService, RobinEngine.class.getMethod("snapshot", Engine.SnapshotHandler.class));...
            bindEngineMethodToCounterAndAction(parfaitService, RobinEngine.class.getMethod("create", Engine.Create.class));
...

This file has been truncated. show original

What I'm looking to do is to 'reach in' to the internals of the Engine so
that I can wrap them up in per-index, even per-shard timing methods for each
type of operations. Eventually this will expose via this plugin to
Performance Co-Pilot and be able to show which indexes/shards even are
consuming CPU(user,kernel), elapsed time, searches (counts and times etc).

I was hoping to use AOP so that I don't need to be invasive. So if there is
other ways to hook in to these internals if you could help me out, that
would be great.

Incidently the Netty nio stuff has bamboozled me, I'm trying to work out how
to 'wrap' the streams so that I can wrap a byte stream counter around them.
Ideally I could then track a counter for the # REST calls (across all
types), how long in elapsed time, and how many bytes are being streamed on a
per-host basis.

If you can help me work out how to reach into the Engine for this AOP like
per-shard, I'll fix it up and show you the plugin so you can see from there
where I want to go.

thanks!

Paul

kimchy · August 17, 2011, 11:49am

Heya,

I think that something like this should be built as a feature in
elasticsearch. Stats collection is something important, and we should have a
module for that, that both reports the stats and also allow for custom stats
"publishing" options. Lets work on getting this into the "core" of
elasticsearch?

Regarding the netty aspect, it should be simple to add, but again with
some internal work. Open an issue for it. That one is a simple one.

On Wed, Aug 17, 2011 at 8:12 AM, Paul Smith tallpsmith@gmail.com wrote:

On 17 August 2011 13:27, Shay Banon kimchy@gmail.com wrote:

elasticsearch has embedded guice 2.0 codebase since it uses a narrow set
of features form guice, and 3.0 is considerably slower than 2.0. As part of
the embedding process, a version guice without AOP was embedded in order to
try and reduce the size of the jar, as well as not to use cglib and the like
for enhancements (they are faster to create, but an overhead loaded classes
wise).

Are you using AOP? If so, why? Is there a chance to simply move that logic
to a specific callback based design (which will be faster)?

Originally, I had something like this:

reason for needing bindInterceptor · GitHub

What I'm looking to do is to 'reach in' to the internals of the Engine so
that I can wrap them up in per-index, even per-shard timing methods for each
type of operations. Eventually this will expose via this plugin to
Performance Co-Pilot and be able to show which indexes/shards even are
consuming CPU(user,kernel), elapsed time, searches (counts and times etc).

I was hoping to use AOP so that I don't need to be invasive. So if there
is other ways to hook in to these internals if you could help me out, that
would be great.

Incidently the Netty nio stuff has bamboozled me, I'm trying to work out
how to 'wrap' the streams so that I can wrap a byte stream counter around
them. Ideally I could then track a counter for the # REST calls (across all
types), how long in elapsed time, and how many bytes are being streamed on a
per-host basis.

If you can help me work out how to reach into the Engine for this AOP like
per-shard, I'll fix it up and show you the plugin so you can see from there
where I want to go.

thanks!

Paul

Paul_Smith · August 17, 2011, 1:00pm

On 17 August 2011 21:49, Shay Banon kimchy@gmail.com wrote:

Heya,

I think that something like this should be built as a feature in
elasticsearch. Stats collection is something important, and we should have a
module for that, that both reports the stats and also allow for custom stats
"publishing" options. Lets work on getting this into the "core" of
elasticsearch?

On the one hand I agree, on the other, there's a million ways to do this.
We have OpenTSDB with it's collector service, BigDesk etc. Lots of people
will have different use cases, and the data being exported by the REST api
is pretty darn useful as a general mechanism.

My embryonic plugin is located here:

https://github.com/Aconex/elasticsearch-pcp

The docs there should work for 0.17.6.

it's not exporting much just yet, and I don't have enough info in the README
file as a mini tutorial yet to show the true power you can do with PCP.
Because the interception bit is 'missing' I can't really show the really
cool bit just yet. If we can 'find a way' to allow this interception back,
then I'll be able to bring that functionality back and demo it in the plugin
form. We can continue to discuss after that about what elasticsearch should
have in it or not.

Regarding the netty aspect, it should be simple to add, but again with
some internal work. Open an issue for it. That one is a simple one.

I've managed to find the NetworkStats object being exposed in the
NetworkService and can use that for export. that's useful, but not the same
as the bandwidth level info I think would be useful.

I'll open an issue. Thanks Shay!

Paul

Paul_Smith · August 17, 2011, 1:08pm

I've managed to find the NetworkStats object being exposed in the
NetworkService and can use that for export. that's useful, but not the same
as the bandwidth level info I think would be useful.

I'll open an issue. Thanks Shay!

Enhance transport stats to include rx and tx counters · Issue #1254 · elastic/elasticsearch · GitHub

kimchy · August 18, 2011, 7:58pm

On Wed, Aug 17, 2011 at 4:00 PM, Paul Smith tallpsmith@gmail.com wrote:

On 17 August 2011 21:49, Shay Banon kimchy@gmail.com wrote:

Heya,

I think that something like this should be built as a feature in
elasticsearch. Stats collection is something important, and we should have a
module for that, that both reports the stats and also allow for custom stats
"publishing" options. Lets work on getting this into the "core" of
elasticsearch?

On the one hand I agree, on the other, there's a million ways to do this.
We have OpenTSDB with it's collector service, BigDesk etc. Lots of people
will have different use cases, and the data being exported by the REST api
is pretty darn useful as a general mechanism.

Right. The idea that I had is there will be internal structures
that accumulate "stats" or provide them as an API. There will also be a way
to register for stats events internally, though I am not sure you need it.

Once we have it as an API, then the plugin can periodically call those stats
API internally, and report them to whatever monitoring system one so
chooses.

For example, if we have am "index stats" module, that keeps count of some
counters (like number of invocations, total time), then the custom monitor
plugin can call this periodically, and report those stats to an externa
system.

My embryonic plugin is located here:

https://github.com/Aconex/elasticsearch-pcp

The docs there should work for 0.17.6.

it's not exporting much just yet, and I don't have enough info in the
README file as a mini tutorial yet to show the true power you can do with
PCP. Because the interception bit is 'missing' I can't really show the
really cool bit just yet. If we can 'find a way' to allow this interception
back, then I'll be able to bring that functionality back and demo it in the
plugin form. We can continue to discuss after that about what elasticsearch
should have in it or not.

Regarding the netty aspect, it should be simple to add, but again with
some internal work. Open an issue for it. That one is a simple one.

I've managed to find the NetworkStats object being exposed in the
NetworkService and can use that for export. that's useful, but not the same
as the bandwidth level info I think would be useful.

I'll open an issue. Thanks Shay!

Paul