Cloudstack Swift River

Elastic Stack Elasticsearch
#1

Hello,

Before developping a CloudStack Swift river
http://docs.openstack.org/trunk/openstack-compute/starter/content/ch05s01s02.html
Do you know if this solution exists ?

It seems that is close to the "dropbox river" solution developed by david ?
I have never developped plugins for ES, only used this search.

Is it possible to develop a plugin based on river CloudStack Swift River
and how difficult and time expected to develop this kind of solution?

Thanks,

--

#2

If cloudstack has a nice API and is easy to use, the elasticsearch river side is
quite easy to understand.
Just fork any river project (RSS, FS, Twitter...) and go for it.

BTW, I wrote something about it (french content):
http://dev.david.pilato.fr/?p=171 http://dev.david.pilato.fr/?p=171

Feel free to ask any question about it.

HTH
David.

Le 10 janvier 2013 à 17:31, Christophe HENRY chrispub12@free.fr a écrit :

Hello,

Before developping a CloudStack Swift river

http://docs.openstack.org/trunk/openstack-compute/starter/content/ch05s01s02.html
Do you know if this solution exists ?

It seems that is close to the "dropbox river" solution developed by david ?
I have never developped plugins for ES, only used this search.

Is it possible to develop a plugin based on river CloudStack Swift River and
how difficult and time expected to develop this kind of solution?

Thanks,

--

--
David Pilato
http://www.scrutmydocs.org/
http://dev.david.pilato.fr/
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

--

#3

Oups, There is a mistake : not read CloudStack but OpenStack.
But the link is good.

Hi David,
If you base your answer with the web link with api , that's good for me :slight_smile:

--

#4

Oups there is a mistake, read OpenStack instead of Cloudstack.

Hi David,
If you base your answer with the web link with api , that's good for me :slight_smile:

Thanks for your answer and the tuto.

--

#5

Can you explain why a river?

From what I understand, OpenStack is a cloud service like Amazon AWS, not
really a source for a river.

There is a cloud-aws plugin, which supersedes the zen discovery (and and
obsoleted S3 shared storage) but that is all I know about ES cloud support
yet.

I'm very enthusiastic about JClouds http://www.jclouds.org

"jclouds tests support of 30 cloud providers and cloud software stacks
including Amazon, GoGrid, Ninefold, vCloud, OpenStack, and Azure."

By managing compute nodes and blob stores, JClouds could be the starting
point for an ES launch tool and a discovery plugin (maybe based on the
zookeeper plugin at https://github.com/sonian/elasticsearch-zookeeper) for
more than 30 cloud infrastructures!

If someone would like to start work on this, it would be very, very
exciting.

My favorite is also OpenStack since everybody can build OpenStack private
clouds because it is open source.

Cheers,

Jörg

--

#6

Sorry, I forgot Apache Whirr http://whirr.apache.org/ where Elasticsearch
deployment on AWS and Rackspace is supported, but it is not clear to me how
JClouds is used.

Jörg

--

#7

Hello Jörg,

I think he stores files (oOo, Office, pdf...) on openstack swift space (a dropbox like feature) and want to index them.
He did not say that he wants to run ES on a openstack cloud.

So, I suppose that a river makes sense here.

My 2 cents.

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 13 janv. 2013 à 10:47, Jörg Prante joergprante@gmail.com a écrit :

Can you explain why a river?

From what I understand, OpenStack is a cloud service like Amazon AWS, not really a source for a river.

There is a cloud-aws plugin, which supersedes the zen discovery (and and obsoleted S3 shared storage) but that is all I know about ES cloud support yet.

I'm very enthusiastic about JClouds http://www.jclouds.org

"jclouds tests support of 30 cloud providers and cloud software stacks including Amazon, GoGrid, Ninefold, vCloud, OpenStack, and Azure."

By managing compute nodes and blob stores, JClouds could be the starting point for an ES launch tool and a discovery plugin (maybe based on the zookeeper plugin at https://github.com/sonian/elasticsearch-zookeeper) for more than 30 cloud infrastructures!

If someone would like to start work on this, it would be very, very exciting.

My favorite is also OpenStack since everybody can build OpenStack private clouds because it is open source.

Cheers,

Jörg

--

#8

Hey Christophe,

Have a look at this nice blog post from Luca: http://blog.trifork.nl/2013/01/10/how-to-write-an-elasticsearch-river-plugin/

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 13 janv. 2013 à 11:00, David Pilato david@pilato.fr a écrit :

Hello Jörg,

I think he stores files (oOo, Office, pdf...) on openstack swift space (a dropbox like feature) and want to index them.
He did not say that he wants to run ES on a openstack cloud.

So, I suppose that a river makes sense here.

My 2 cents.

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 13 janv. 2013 à 10:47, Jörg Prante joergprante@gmail.com a écrit :

Can you explain why a river?

From what I understand, OpenStack is a cloud service like Amazon AWS, not really a source for a river.

There is a cloud-aws plugin, which supersedes the zen discovery (and and obsoleted S3 shared storage) but that is all I know about ES cloud support yet.

I'm very enthusiastic about JClouds http://www.jclouds.org

"jclouds tests support of 30 cloud providers and cloud software stacks including Amazon, GoGrid, Ninefold, vCloud, OpenStack, and Azure."

By managing compute nodes and blob stores, JClouds could be the starting point for an ES launch tool and a discovery plugin (maybe based on the zookeeper plugin at https://github.com/sonian/elasticsearch-zookeeper) for more than 30 cloud infrastructures!

If someone would like to start work on this, it would be very, very exciting.

My favorite is also OpenStack since everybody can build OpenStack private clouds because it is open source.

Cheers,

Jörg

--

--

#9

Thanks Jörg and David for your contribution.

This is exactly what explains David.
My need is to collect the documents (pdf, office, ...) stored in containers
swift and index content in Elasticsearch.

Indeed, it does not seem very complicated to write a river. Thank you for
this link.

--

#10