Java API: Transforming, chaining ListenableActionFuture is tricky at best. How about extending Guava ListenableFuture to benefit from the featureset it provides?

Stephane_Bastian · April 17, 2013, 7:21am

Hi,

I've been using ES for almost 2 years now and I'm very pleased with it. I
am especially happy with the fact that everything is Asynchronous.

However, the one thing that sometimes gets in the way is when the code
calling ES has some logic to transform, chain or compose several
ListenableActionFuture.

Outside ES, we are using google Guava to solve these tricky problems and
make extensive use of Futures.transform (to transform a ListenableFuture in
another ListenableFuture) or Futures.allAsList, etc.
For people that have no clue of what I am talking about ->

ListenableActionFuture returned by ES are great but there are no helper
methods to transform, chain, group ListenableActionFutures.

So the question is:
How about modifying ListenableActionFuture so that it extends
ListenableFuture from Guava? This would allow for super simple integration
with Guava concurrent package.

Another question for my own curiosity:
Do you use google guava with ES?

Cheers!

Stephane

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Ivan · April 17, 2013, 5:42pm

After having used Scala, I love the power and flexibility of using Futures.
I have not had a use case to do any transformations with Elasticsearch
futures, but it would be interesting to see.

Elasticsearch uses Guava for the field caches. Not sure if there are any
other uses.

--
Ivan

On Wed, Apr 17, 2013 at 12:21 AM, Stephane Bastian <
stephane.bastian.dev@gmail.com> wrote:

Hi,

I've been using ES for almost 2 years now and I'm very pleased with it. I
am especially happy with the fact that everything is Asynchronous.

However, the one thing that sometimes gets in the way is when the code
calling ES has some logic to transform, chain or compose several
ListenableActionFuture.

Outside ES, we are using google Guava to solve these tricky problems and
make extensive use of Futures.transform (to transform a ListenableFuture in
another ListenableFuture) or Futures.allAsList, etc.
For people that have no clue of what I am talking about ->
GitHub - google/guava: Google core libraries for Java

ListenableActionFuture returned by ES are great but there are no helper
methods to transform, chain, group ListenableActionFutures.

So the question is:
How about modifying ListenableActionFuture so that it extends
ListenableFuture from Guava? This would allow for super simple integration
with Guava concurrent package.

Another question for my own curiosity:
Do you use google guava with ES?

Cheers!

Stephane

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

jprante · April 17, 2013, 7:09pm

You might want to hack ES, the interface is
org.elasticsearch.action.ActionFuture. It inherits from
java.util.concurrent.Future, and since Guava's ListenableActionFuture
also inherits from it, it should be possible to replace the JDK Future
with the Guava Future.

Guava is used all over the place. com.google.common is shaded into
org.elasticsearch.common which makes it a bit hard to trace the exact usage.

I use Guava with ES, only the collection helpers yet. Would love to
explore the performance of Guava's EventBus for publishing event streams
in a node, transported by websockets to other nodes, to subscribed clients.

Jörg

Am 17.04.13 09:21, schrieb Stephane Bastian:

Hi,

I've been using ES for almost 2 years now and I'm very pleased with
it. I am especially happy with the fact that everything is Asynchronous.

However, the one thing that sometimes gets in the way is when the code
calling ES has some logic to transform, chain or compose several
ListenableActionFuture.

Outside ES, we are using google Guava to solve these tricky problems
and make extensive use of Futures.transform (to transform a
ListenableFuture in another ListenableFuture) or Futures.allAsList, etc.
For people that have no clue of what I am talking about ->
GitHub - google/guava: Google core libraries for Java

ListenableActionFuture returned by ES are great but there are no
helper methods to transform, chain, group ListenableActionFutures.

So the question is:
How about modifying ListenableActionFuture so that it extends
ListenableFuture from Guava? This would allow for super simple
integration with Guava concurrent package.

Another question for my own curiosity:
Do you use google guava with ES?

Cheers!

Stephane

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Stephane_Bastian · April 17, 2013, 7:57pm

Hi Ivan,

We mainly have 2 uses cases to wrap ES ListenableActionFuture:

Use case 1:

We have built a library that maps entities and their relationships to ES.
It is very similar to JPA, the main difference is that the target is ES and
not a relational db. Also all methods are async and return a guava
ListenableFuture.

As an example, lets take the method that returns a single entity. Here is
its signature:
public ListenableFuture get(Class cls, String id);

Internally this method does the following:

Calls client.prepareGet(index, type, id).execute(); --> this returns a
ListenableActionFuture
Registers a listener on the listenableActionFuture so itś non blocking
When the listener is called we convert the json to the correct Java
object(along with its relationships eagerly or lazily loaded if we asked
for it, but that is a different story)

This code is working fine but it would be much easier if ES
ListenableActionFuture could subclass guava ListenableFuture.
In essence we would then simply use guava Futures.transform(xx, xx) that
applies a function to the ListenableFuture to transform the json to a java
object
Less boilerplate code to write.

Use case 2:

In some other part of our application, we call ElasticSearch (lets say a
simple get by id), then we call another service containing extra data for
the same id.
Both calls are async. When we get both result, we simply need to stuff
the json of the second call into the json we got from ES. of course the
result is itself a ListenableFuture.
Once again here, it's quite a bit of code that's not easy to read, to make
it work.

In both cases, it's fairly painful to transform/chain ES
ListenableActionFuture.

Concerning guava, you are right, ES already uses it. It inlines some(all?)
google guava code. Actually I believe that quite a bit of the concurrent
code comes from guava. So subclassing guava ListenableFuture should not be
a problem. But this will of course make the internal guava ListenableFuture
part of the ES api..

To summarize, I believe that there is a need to make ES
ListenableActionFuture easily chainable/transformable.
It can be achieved by

subclassing guava ListenableFuture
converting ES ListenableActionFuture to guava ListenableFuture (although
this one seems trickier because guava allows listeners to run different
threads)

Stephane

On Wednesday, April 17, 2013 7:42:59 PM UTC+2, Ivan Brusic wrote:

After having used Scala, I love the power and flexibility of using

Futures. I have not had a use case to do any transformations with
ElasticSearch futures, but it would be interesting to see.
ElasticSearch uses Guava for the field caches. Not sure if there are
any other uses.
--
Ivan
On Wed, Apr 17, 2013 at 12:21 AM, Stephane Bastian
stephane.b...@gmail.com wrote:

    Hi,

    I've been using ES for almost 2 years now and I'm very pleased with

it. I am especially happy with the fact that everything is Asynchronous.

    However, the one thing that sometimes gets in the way is when the

code calling ES has some logic to transform, chain or compose several
ListenableActionFuture.

    Outside ES, we are using google Guava to solve these tricky

problems and make extensive use of Futures.transform (to transform a
ListenableFuture in another ListenableFuture) or Futures.allAsList, etc.
For people that have no clue of what I am talking about ->

    ListenableActionFuture returned by ES are great but there are no

helper methods to transform, chain, group ListenableActionFutures.

    So the question is:
    How about modifying ListenableActionFuture so that it extends

ListenableFuture from Guava? This would allow for super simple integration
with Guava concurrent package.

    Another question for my own curiosity:
    Do you use google guava with ES?

    Cheers!

    Stephane





    -- 
    You received this message because you are subscribed to the Google

Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to elasticsearc...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Stephane_Bastian · April 17, 2013, 8:06pm

Hey Jorg,

You are right, guava provides a helper to convert a Future in a
ListenableFuture. It's the class JdkFutureAdapters.listenInPoolThread()http://docs.guava-libraries.googlecode.com/git/javadoc/com/google/common/util/concurrent/JdkFutureAdapters.html#listenInPoolThread(java.util.concurrent.Future).

However I read that this method is far from ideal and not very scalable:
Quote from http://java.dzone.com/articles/listenablefuture-guava

If you are stuck with some legacy API that still returns Future you may try

JdkFutureAdapters.listenInPoolThread()http://docs.guava-libraries.googlecode.com/git/javadoc/com/google/common/util/concurrent/JdkFutureAdapters.html#listenInPoolThread(java.util.concurrent.Future)which is an adapter converting plain
Future to ListenableFuture. But keep in mind that once you start
using addListener(), each such adapter will require one thread
exclusively to work so this solution doesn't scale at all and you should
avoid it.

Stephane

On Wednesday, April 17, 2013 9:09:56 PM UTC+2, Jörg Prante wrote:

You might want to hack ES, the interface is
org.elasticsearch.action.ActionFuture. It inherits from
java.util.concurrent.Future, and since Guava's ListenableActionFuture
also inherits from it, it should be possible to replace the JDK Future
with the Guava Future.

Guava is used all over the place. com.google.common is shaded into
org.elasticsearch.common which makes it a bit hard to trace the exact
usage.

I use Guava with ES, only the collection helpers yet. Would love to
explore the performance of Guava's EventBus for publishing event streams
in a node, transported by websockets to other nodes, to subscribed
clients.

Jörg

Am 17.04.13 09:21, schrieb Stephane Bastian:

Hi,

I've been using ES for almost 2 years now and I'm very pleased with
it. I am especially happy with the fact that everything is Asynchronous.

However, the one thing that sometimes gets in the way is when the code
calling ES has some logic to transform, chain or compose several
ListenableActionFuture.

Outside ES, we are using google Guava to solve these tricky problems
and make extensive use of Futures.transform (to transform a
ListenableFuture in another ListenableFuture) or Futures.allAsList, etc.
For people that have no clue of what I am talking about ->
GitHub - google/guava: Google core libraries for Java

ListenableActionFuture returned by ES are great but there are no
helper methods to transform, chain, group ListenableActionFutures.

So the question is:
How about modifying ListenableActionFuture so that it extends
ListenableFuture from Guava? This would allow for super simple
integration with Guava concurrent package.

Another question for my own curiosity:
Do you use google guava with ES?

Cheers!

Stephane

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Stephane_Bastian · April 19, 2013, 3:14pm

Hey,

I found a simple solution to the problem I described above. In case someone
comes across the same issue here it is the solution which is simply to
convert an ES ListenableActionFuture to a guava ListenableFuture.
Here is the code:

public final static <T> ListenableFuture<T>

esListenableToGuavaListenable(ListenableActionFuture
listenableActionFuture) {
final SettableFuture result = SettableFuture.create();
listenableActionFuture.addListener(new ActionListener() {
@Override
public void onResponse(T response) {
result.set(response);
}
        @Override
        public void onFailure(Throwable e) {
            result.setException(e);
        }
    });
    return result;
}

I then simply call this method each time ES returns a
ListenableActionFuture, basically each time you call .execute() on a builder

Hope this helps,

Stephane

Le mercredi 17 avril 2013 22:06:34 UTC+2, Stephane Bastian a écrit :

Hey Jorg,

You are right, guava provides a helper to convert a Future in a
ListenableFuture. It's the class JdkFutureAdapters.listenInPoolThread()http://docs.guava-libraries.googlecode.com/git/javadoc/com/google/common/util/concurrent/JdkFutureAdapters.html#listenInPoolThread(java.util.concurrent.Future).

However I read that this method is far from ideal and not very scalable:
Quote from http://java.dzone.com/articles/listenablefuture-guava

If you are stuck with some legacy API that still returns Future you may

try JdkFutureAdapters.listenInPoolThread()http://docs.guava-libraries.googlecode.com/git/javadoc/com/google/common/util/concurrent/JdkFutureAdapters.html#listenInPoolThread(java.util.concurrent.Future)which is an adapter converting plain
Future to ListenableFuture. But keep in mind that once you start
using addListener(), each such adapter will require one thread
exclusively to work so this solution doesn't scale at all and you should
avoid it.

Stephane

On Wednesday, April 17, 2013 9:09:56 PM UTC+2, Jörg Prante wrote:

You might want to hack ES, the interface is
org.elasticsearch.action.ActionFuture. It inherits from
java.util.concurrent.Future, and since Guava's ListenableActionFuture
also inherits from it, it should be possible to replace the JDK Future
with the Guava Future.

Guava is used all over the place. com.google.common is shaded into
org.elasticsearch.common which makes it a bit hard to trace the exact
usage.

I use Guava with ES, only the collection helpers yet. Would love to
explore the performance of Guava's EventBus for publishing event streams
in a node, transported by websockets to other nodes, to subscribed
clients.

Jörg

Am 17.04.13 09:21, schrieb Stephane Bastian:

Hi,

I've been using ES for almost 2 years now and I'm very pleased with
it. I am especially happy with the fact that everything is
Asynchronous.

However, the one thing that sometimes gets in the way is when the code
calling ES has some logic to transform, chain or compose several
ListenableActionFuture.

Outside ES, we are using google Guava to solve these tricky problems
and make extensive use of Futures.transform (to transform a
ListenableFuture in another ListenableFuture) or Futures.allAsList,
etc.
For people that have no clue of what I am talking about ->
GitHub - google/guava: Google core libraries for Java

ListenableActionFuture returned by ES are great but there are no
helper methods to transform, chain, group ListenableActionFutures.

So the question is:
How about modifying ListenableActionFuture so that it extends
ListenableFuture from Guava? This would allow for super simple
integration with Guava concurrent package.

Another question for my own curiosity:
Do you use google guava with ES?

Cheers!

Stephane

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Bruce_Ritchie · April 22, 2013, 1:10pm

The source tells all, all it takes is for you to take a quick look.

The pom.xml in github:

    <dependency>
        <groupId>com.google.guava</groupId>
        <artifactId>guava</artifactId>
        <version>14.0.1</version>
        <scope>compile</scope>
        <exclusions>
            <exclusion>
                <groupId>com.google.code.findbugs</groupId>
                <artifactId>jsr305</artifactId>
            </exclusion>
        </exclusions>
    </dependency>

On Wednesday, April 17, 2013 3:21:51 AM UTC-4, Stephane Bastian wrote:

Another question for my own curiosity:
Do you use google guava with ES?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Stephane_Bastian · April 22, 2013, 2:25pm

Hi Bruce,

I'm looking at the question I asked and it was fairly confusing. The fact
that ES uses guava internally wasn't the question. Sorry about that.

The question should have been:
Do you use google guava ListenableFuture with ES? If yes, how?

Le lundi 22 avril 2013 15:10:03 UTC+2, Bruce Ritchie a écrit :

The source tells all, all it takes is for you to take a quick look.

The pom.xml in github:
    <dependency>
        <groupId>com.google.guava</groupId>
        <artifactId>guava</artifactId>
        <version>14.0.1</version>
        <scope>compile</scope>
        <exclusions>
            <exclusion>
                <groupId>com.google.code.findbugs</groupId>
                <artifactId>jsr305</artifactId>
            </exclusion>
        </exclusions>
    </dependency>
On Wednesday, April 17, 2013 3:21:51 AM UTC-4, Stephane Bastian wrote:

Another question for my own curiosity:
Do you use google guava with ES?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.