Jclouds provider

Elastic Stack Elasticsearch
1

What happened to the jclouds provider that was in ES pre-0.8? I'm
facing deploying ES to RackSpace cloud or SoftLayer and would like to
use my cloud providers blob storage and discovery.

Is there still a plugin available for jclouds or for cloud providers
other than AWS?

Thanks,

Josh

2
3

I know that doesn't answer your question directly, but Shay dropped jClouds
because it was having some problems early in its life. I agree that it would
be nice to take another look at it.

4

The issue mentions that RackSpace is supported but I don't see any
documentation or reference to this support. Is it hiding somewhere? :slight_smile:

@James, nice AWS post too. Helps to describe how ES should integrate
with clouds along with providing easy to follow steps for getting
started on AWS.

On Sep 9, 3:51 pm, James Cook jc...@tracermedia.com wrote:

I know that doesn't answer your question directly, but Shay dropped jClouds
because it was having some problems early in its life. I agree that it would
be nice to take another look at it.

5

Thanks, I'm glad it helped.

No, Rackspace is not supported. The idea was that it was "supported" back in
pre 0.8 builds because jClouds was used over the current cloud discovery
process which only supports EC2 currently.

Since Shay has externalized the cloud discovery process and gateway support
for AWS specifically, it shouldn't be too hard to change it back over to
jClouds support. I'm sure he would appreciate a pull request. I wish I had
the time to work on it, because AWS is $$$, and I also would like other
alternatives.

The key would be to make sure that jClouds supports the necessary contracts
for discovery (discover other nodes and provide region, group, tags), and
the appropriate methods to support the cloud-based gateway (which I am
unsure offhand how this works, i.e. sync/async transfer, auto chunking?,
reading)

6

We are in the same boat looking at growing AWS monthly bills :wink:

I will file a ticket and also if I have time, work on an
implementation and a pull request. Would be great to get Shay's input
as to why the jclouds-based integration was backed out, what sorts of
problems he had, etc. At the very least, basic RackSpace support
through their API should be possible.

See: https://github.com/elasticsearch/elasticsearch/issues/1319

7

This might explain a bit:

8

Heya,

First, a quick explanation of what the aws cloud plugin provides.
Basically, it provides two main features:

  1. Automatic discovery based on cloud API. This is a nice extension to the
    zen discovery module, where the unicast list of hosts is built automatically
    using the AWS API. It would be nice to have it using jclouds or directly
    using other cloud APIs. The problem with jclouds back then was that it was
    doing far too much in order to just get the current running instances. Need
    to check if it improved.

    But, and this is important, you don't have to use it. You can build your
    own unicast discovery list and use it on any cloud provider or environment
    that does not support multicast. The lack of support for rackspace does
    not     mean you can't run elasticsearch on rackspace.

  2. If the cloud provider has support for blob store, then a blob store
    gateway can be implemented. This is again not necessarily required. You can
    run elasticsearch with the default local gateway which will work.

Mentioning it again: both extensions, the auto disco based on the cloud API
And blob store gateway are nice, but are not required in order to run
elasticsearch on cloud providers. Without them, elasticsearch is similar to
other "nosql" solutions, or even mysql (and the like) in terms of how you
deploy it on your favorite cloud. Of course, it would be nice to have those
extensions available to simplify even more the deployment on other clouds.

On Fri, Sep 9, 2011 at 5:14 PM, James Cook jcook@tracermedia.com wrote:

This might explain a bit:
Cloud Discovery does not appear to work · Issue #197 · elastic/elasticsearch · GitHub

9

Shay, thanks for the notes. Good to know the problems with jclouds in
the past. I'm definitely more interested in getting the discovery and
blob storage working as an optimization. I think it's clear that
unicast discovery+local gateway works on any cloud provider.

I hope to have some time to look into at least an initial jclouds
solution, but could take some months given that it's an
optimization :wink:

Cheers,

Josh

10

Josh,

I would be really interested in helping with the jclouds work, if I
can. I work with the jclouds founder, Adrian, so that will hopefully
make it easier to sort out any integration issues with jclouds and the
blob store if they occur :wink: I think the 1.1.1 release of jclouds is
much improved, and as mentioned above it would greatly simplify
deployment to a new cloud provider.

As for 'doing too much to get the current running instances', I'm not
sure what was happening there. It should be a pretty simple task in
jclouds...

Cheers,
Andrew.

On Sep 12, 9:27 am, Josh Devins i...@joshdevins.net wrote:

Shay, thanks for the notes. Good to know the problems withjcloudsin
the past. I'm definitely more interested in getting the discovery and
blob storage working as an optimization. I think it's clear that
unicast discovery+local gateway works on any cloud provider.

I hope to have some time to look into at least an initialjclouds
solution, but could take some months given that it's an
optimization :wink:

Cheers,

Josh

11

Hey Andrew,

(As I mentioned in my email earlier) I haven't looked into starting
the jclouds integration yet but probably will in the next 6-weeks. Let
me know if you get started
before that! Will update the ticket as I go.

I agree that in principle/theory it should not be too complicated, but
practice will always have some gotcha's! Hopefully any issues that
arise can be dealt with my you/Adrian/jclouds. If you have time to
work on this before I get to it, that would be great. It's low on my
priority list right now since, as Shay mentioned, this is an
optimization. You can still use local storage and unicast discovery on
any cloud provider/DC/network setup.

Josh

12

My brief glance at the API seemed to indicate that the concept of tags in
(i.e. EC2) is not supported. Groups are supported.