Shay,
Unfortunately, Azure doesn’t have something like EBS (where you can mount a drive if you are in the local compute cluster).
You have access to local storage, but if your app is recycled, then you lose that.
If you want persistent storage then you have to go through the Azure blob.
From: Shay Banon [mailto:shay.banon@elasticsearch.com]
Sent: Tuesday, March 01, 2011 6:44 PM
To: users@elasticsearch.com
Subject: Re: RE: RE: RE: RE: Cloud plugin for 0.15.0 (using Azure blob storage)
Not familiar with Azure, but if it has something similar to AWS EBS, you can use local gateway and have the data location stored on it.
On Wednesday, March 2, 2011 at 12:21 AM, Administrator wrote:
Shay,
I’m not sure what you mean by local gateway?
What I’m trying to do right now is a proof-of-concept; if I can get a locally running instance of ES to connect to Azure, then getting it to run in an Azure Compute instance is easy enough and I’d know for sure that I would have persistent storage for ES.
I’d only have one compute instance running, so I wouldn’t need discovery of other nodes. Exposing the port in Azure is easy enough, there is a config setting where you can map the port, I’d just map the port of my choosing to the ES instance port.
- Nick
From: Shay Banon [mailto:shay.banon@elasticsearch.com]
Sent: Tuesday, March 01, 2011 4:54 PM
To: users@elasticsearch.com
Subject: Re: RE: RE: RE: Cloud plugin for 0.15.0 (using Azure blob storage)
You can use local gateway for now, no?
On Tuesday, March 1, 2011 at 9:57 PM, Administrator wrote:
Shay,
Here’s the issue:
And I’m serious about the Azure account, all I’d have to do is give you the account name and key. I can regenerate the key once done and never have to give you credentials.
From what I can tell, this is the only thing keeping me from using ElasticSearch right now. =)
Much thanks for even entertaining the idea.
- Nick
From: Shay Banon [mailto:shay.banon@elasticsearch.com]
Sent: Tuesday, March 01, 2011 2:22 PM
To: users@elasticsearch.com
Subject: Re: RE: RE: Cloud plugin for 0.15.0 (using Azure blob storage)
You can't get that plugin anymore, since it was removed :). Open a feature request in the github issues and we can track it there.
-shay.banon
On Tuesday, March 1, 2011 at 9:20 PM, Administrator wrote:
Shay,
Where can I get that plugin?
Also, where can I make a request for Azure blob support? There is an SDK from MS for Azure blob support as well, so it wouldn’t have to be written from scratch.
I’d even be willing to loan you my azure blob account for testing if you were to take up the cause. =)
- Nick
From: Shay Banon [mailto:shay.banon@elasticsearch.com]
Sent: Tuesday, March 01, 2011 2:07 PM
To: users@elasticsearch.com
Subject: Re: RE: Cloud plugin for 0.15.0 (using Azure blob storage)
There use to be a generic cloud plugin that used jclouds to work with any cloud provider that jclouds supports. It has been removed because jclouds was not mature enough at the time, and replaced with the cloud-aws plugin that includes the formal AWS sdk. jclouds has progressed a lot since then, so it might make sense to give it another try and have a generic cloud plugin that uses it.
On Tuesday, March 1, 2011 at 9:03 PM, Administrator wrote:
Shay,
I’m slightly confused. In a thread from May 2010 (http://elasticsearch-users.115913.n3.nabble.com/Just-Pushed-Cloud-Plugin-Using-the-cloud-for-gateway-storage-and-for-auto-discovery-td784890.html) you state in the second paragraph:
The first support is allowing to use the cloud as a gateway storage. This means that Amazon S3, Rackspace CloudFiles, or Azure Blob can be used to store both the cluster meta data and each index information (index files and transaction logs).
You specifically mention S3, Rackspace, and Azure Blob.
Or was it meant as there is potential for support for Azure?
If that’s the case, how can one best make a suggestion to implement support? There are a number of people who are deploying Java applications/servers on Azure (MS is putting out support for Tomcat in Azure, and there are a number of demos of running Jetty as well) and I’d like to be able to let .NET developers expose ES through Azure as well.
Quite frankly, it’s because the support for search is so bad in the .NET space (yes, Lucene.NET, and I’ve worked with that, but I really need something like ES at this point) that I’m more than happy to interop and in this case, ES gives me everything I need and so much more.
- Nick
From: Shay Banon [mailto:shay.banon@elasticsearch.com]
Sent: Tuesday, March 01, 2011 1:48 PM
To: users@elasticsearch.com
Subject: Re: Cloud plugin for 0.15.0 (using Azure blob storage)
There ins't support for Azure blob store, only for AWS at the moment.
On Tuesday, March 1, 2011 at 5:10 PM, Administrator wrote:
I'm simply trying to run ElasticSearch on my local machine, using
Azure blob storage as a backing store.
I've downloaded ES 0.15.0, and unzipped it into a directory.
I then changed the config file like so:
cluster:
name: OneFrameLink
cloud:
account:
key:
blobstore:
type: Azureblob
gateway:
type: cloud
cloud:
container: data
I've populated account with the name of the storage account in
Azure, and the key with the primary secret key. I've connected to the
account through CloudXplorer and verified that the account does exist, and
created the empty container "data".
The thing is, when I run elasticsearch.bat from the command line, I
get the following:
[2011-03-01 09:49:38,162][INFO ][node] [Barbarus]
{elasticsearch/0.15.0}[9144]: initializing ...
[2011-03-01 09:49:38,165][INFO ][plugins] [Barbarus] loaded []
[2011-03-01 09:49:38,184][ERROR][bootstrap] {elasticsearch/0.15.0}:
Initialization Failed ...
- NoClassSettingsException[Failed to load class setting [gateway.type] with
value [cloud]]
ClassNotFoundException[cloud]
Searching on Google brings me here:
http://elasticsearch-users.115913.n3.nabble.com/Weird-Startup-Problem-on-14-
Snapshot-td1947233.html
Fortunately, I am running elasticsearch.bat from within the bin
directory (not from the root and specifying bin/.. as Shay says).
However, it seems I don't have the cloud plugin. Ok, so I then run:
plugin.bat -install cloud
But that gives me:
-> Installing cloud Downloading plugin from
http://elasticsearch.googlecode.com/svn/plugins/cloud/elasticsearch-cloud-0.
15.0.zip
Failed to install cloud, reason: Can't get
http://elasticsearch.googlecode.com/svn/plugins/cloud/elasticsearch-cloud-0.
15.0.zip to C:\Users<name>\Desktop\elasticsearch-0.15.0\plugins\cloud.zip
Doing a Google search on "elasticsearch cloud plugin", I notice this
entry:
http://code.google.com/p/elasticsearch/source/browse/plugins/cloud-aws/elast
icsearch-cloud-aws-0.13.1.zip?spec=svn20&r=20
Which seems to be an AWS-specific version of the cloud. Navigating
through the svn, I eventually find this:
http://code.google.com/p/elasticsearch/source/browse/plugins/?r=20#plugins%2
Fcloud
And there appears that 0.8.0 is the latest version of the
elasticsearch cloud plugin. Ok, so I download that into the plugin
directory and run ES, and I get the same ClassNotFoundException.
I unzip the file and place the contents in the plugins directory,
run ES and I get the same ClassNotFoundException.
So the most pertinent question here is, how do I get the cloud
plugin so I can even ^attempt^ to use the cloud plugin and use that as my
data store when running from my local machine (note, this will eventually be
pushed to Azure, but I am doing this as a sanity test to see that ES can
access Azure blob storage).
Also, are the config settings as I have them above correct? I know
that once this is deployed to Azure, I will have to handle discovery, but
that's not much of an issue, since the WorkerRole which will run
ElasticSearch can forward a tcp port, which is what I will do for
ElasticSearch (or configure discovery).
Thanks in advance for any help.