Problems with Hadoop plugin

XCouly · September 18, 2012, 2:59pm

I tried to use a Hadoop Gateway on ES 19.9. Every time I tried to download
Hadoop Gateway it's been installing in _site configuration.

When I tried to configure Hadoop gateway with this kind of configuration in
my elasticsearch.yml:

gateway.type : hdfs

gateway.hdfs.uri : hdfs://:8020

gateway.hdfs.path : /user/elasticsearch

it result on an error like this:

jvm 1 | [2012-09-18 16:49:03,355][INFO ][node ]
[ESNodeReader] {0.19.9}[32462]: initializing ...

jvm 1 | [2012-09-18 16:49:03,403][INFO ][plugins ]
[ESNodeReader] loaded [], sites [hadoop]

jvm 1 | [2012-09-18 16:49:03,932][ERROR][bootstrap ]
{0.19.9}: Initialization Failed ...

jvm 1 | - NoClassSettingsException[Failed to load class setting
[gateway.type] with value [hdfs]]

jvm 1 | ClassNotFoundException[hdfs]

So I tried to download and build the plugin to understand how it work and
how I can fix the _site configuration. I succeed in build it with Maeven,
but itch time I test my build I obtain the same error than before:

Running test.java.org.elasticsearch.hadoop.gateway.HdfsGatewayTests

Sep 18, 2012 4:38:10 PM org.elasticsearch.node

INFO: [node1] {0.19.0}[29968]: initializing ...

Sep 18, 2012 4:38:10 PM org.elasticsearch.plugins

INFO: [node1] loaded [hadoop], sites []

Tests run: 5, Failures: 2, Errors: 0, Skipped: 3, Time elapsed: 0.783 sec
<<< FAILURE!

Results:

Failed tests:
setUpNodes(test.java.org.elasticsearch.hadoop.gateway.HdfsGatewayTests):
Failed to load class setting [gateway.type] with value [hdfs]

setUpNodes(test.java.org.elasticsearch.hadoop.gateway.HdfsGatewayTests):
Failed to load class setting [gateway.type] with value [hdfs]

Tests run: 5, Failures: 2, Errors: 0, Skipped: 3

Is there another way to build the plugin or a way to fix the Failed to load
hdfs class? I tried to reference the library instated of just hdfs but I
have the same result.

Thanks for your helps.

--

kimchy · September 18, 2012, 7:52pm

How do you install the hadoop plugin? Btw, the shared gateway configuration is not recommended and the local gateway is the recommended gateway to use (its the default). See the previous mails here on what needs to be done with the local gateway to improve the "backup / restore" aspect to it.

On Sep 18, 2012, at 4:59 PM, XCouly xavier.couly@gmail.com wrote:

I tried to use a Hadoop Gateway on ES 19.9. Every time I tried to download Hadoop Gateway it's been installing in _site configuration.

When I tried to configure Hadoop gateway with this kind of configuration in my elasticsearch.yml:

gateway.type : hdfs
gateway.hdfs.uri : hdfs://:8020
gateway.hdfs.path : /user/elasticsearch

it result on an error like this:

jvm 1 | [2012-09-18 16:49:03,355][INFO ][node ] [ESNodeReader] {0.19.9}[32462]: initializing ...
jvm 1 | [2012-09-18 16:49:03,403][INFO ][plugins ] [ESNodeReader] loaded , sites [hadoop]
jvm 1 | [2012-09-18 16:49:03,932][ERROR][bootstrap ] {0.19.9}: Initialization Failed ...
jvm 1 | - NoClassSettingsException[Failed to load class setting [gateway.type] with value [hdfs]]
jvm 1 | ClassNotFoundException[hdfs]

So I tried to download and build the plugin to understand how it work and how I can fix the _site configuration. I succeed in build it with Maeven, but itch time I test my build I obtain the same error than before:

Running test.java.org.elasticsearch.hadoop.gateway.HdfsGatewayTests
Sep 18, 2012 4:38:10 PM org.elasticsearch.node
INFO: [node1] {0.19.0}[29968]: initializing ...
Sep 18, 2012 4:38:10 PM org.elasticsearch.plugins
INFO: [node1] loaded [hadoop], sites
Tests run: 5, Failures: 2, Errors: 0, Skipped: 3, Time elapsed: 0.783 sec <<< FAILURE!

Results:

Failed tests: setUpNodes(test.java.org.elasticsearch.hadoop.gateway.HdfsGatewayTests): Failed to load class setting [gateway.type] with value [hdfs]
setUpNodes(test.java.org.elasticsearch.hadoop.gateway.HdfsGatewayTests): Failed to load class setting [gateway.type] with value [hdfs]

Tests run: 5, Failures: 2, Errors: 0, Skipped: 3

Is there another way to build the plugin or a way to fix the Failed to load hdfs class? I tried to reference the library instated of just hdfs but I have the same result.

Thanks for your helps.

--

--

WhatElse · September 18, 2012, 8:21pm

I don't know if I'm wrong or not but which advantage brings Hadoop for
Elasticsearch ?

Le mardi 18 septembre 2012 15:52:06 UTC-4, kimchy a écrit :

How do you install the hadoop plugin? Btw, the shared gateway
configuration is not recommended and the local gateway is the recommended
gateway to use (its the default). See the previous mails here on what needs
to be done with the local gateway to improve the "backup / restore" aspect
to it.

On Sep 18, 2012, at 4:59 PM, XCouly <xavier...@gmail.com <javascript:>>
wrote:

I tried to use a Hadoop Gateway on ES 19.9. Every time I tried to download
Hadoop Gateway it's been installing in _site configuration.

When I tried to configure Hadoop gateway with this kind of configuration
in my elasticsearch.yml:

gateway.type : hdfs

gateway.hdfs.uri : hdfs://:8020

gateway.hdfs.path : /user/elasticsearch

it result on an error like this:

jvm 1 | [2012-09-18 16:49:03,355][INFO ][node ]
[ESNodeReader] {0.19.9}[32462]: initializing ...

jvm 1 | [2012-09-18 16:49:03,403][INFO ][plugins ]
[ESNodeReader] loaded , sites [hadoop]

jvm 1 | [2012-09-18 16:49:03,932][ERROR][bootstrap ]
{0.19.9}: Initialization Failed ...

jvm 1 | - NoClassSettingsException[Failed to load class setting
[gateway.type] with value [hdfs]]

jvm 1 | ClassNotFoundException[hdfs]

So I tried to download and build the plugin to understand how it work and
how I can fix the _site configuration. I succeed in build it with Maeven,
but itch time I test my build I obtain the same error than before:

Running test.java.org.elasticsearch.hadoop.gateway.HdfsGatewayTests

Sep 18, 2012 4:38:10 PM org.elasticsearch.node

INFO: [node1] {0.19.0}[29968]: initializing ...

Sep 18, 2012 4:38:10 PM org.elasticsearch.plugins

INFO: [node1] loaded [hadoop], sites

Tests run: 5, Failures: 2, Errors: 0, Skipped: 3, Time elapsed: 0.783 sec
<<< FAILURE!

Results:

Failed tests:
setUpNodes(test.java.org.elasticsearch.hadoop.gateway.HdfsGatewayTests):
Failed to load class setting [gateway.type] with value [hdfs]

setUpNodes(test.java.org.elasticsearch.hadoop.gateway.HdfsGatewayTests):
Failed to load class setting [gateway.type] with value [hdfs]

Tests run: 5, Failures: 2, Errors: 0, Skipped: 3

Is there another way to build the plugin or a way to fix the Failed to
load hdfs class? I tried to reference the library instated of just hdfs but
I have the same result.

Thanks for your helps.

--

--

jprante · September 18, 2012, 10:44pm

Hadoop was designed for large datasets (TB to PB) with rock solid
reliability. You can use Hadoop as a long term storage for Elasticsearch
indexes and metadata (via the gateway mechanism).

Another method for using Hadoop is loading data with map/reduce into
Elasticsearch, for example
see http://thedatachef.blogspot.de/2011/01/bulk-indexing-with-elasticsearch-and.html
or You know, for search. Querying 24 Billion Documents in 900ms | PPT

Jörg

On Tuesday, September 18, 2012 10:21:08 PM UTC+2, WhatElse wrote:

I don't know if I'm wrong or not but which advantage brings Hadoop for
Elasticsearch ?

Le mardi 18 septembre 2012 15:52:06 UTC-4, kimchy a écrit :

How do you install the hadoop plugin? Btw, the shared gateway
configuration is not recommended and the local gateway is the recommended
gateway to use (its the default). See the previous mails here on what needs
to be done with the local gateway to improve the "backup / restore" aspect
to it.

On Sep 18, 2012, at 4:59 PM, XCouly xavier...@gmail.com wrote:

I tried to use a Hadoop Gateway on ES 19.9. Every time I tried to
download Hadoop Gateway it's been installing in _site configuration.

When I tried to configure Hadoop gateway with this kind of configuration
in my elasticsearch.yml:

gateway.type : hdfs

gateway.hdfs.uri : hdfs://:8020

gateway.hdfs.path : /user/elasticsearch

it result on an error like this:

jvm 1 | [2012-09-18 16:49:03,355][INFO ][node ]
[ESNodeReader] {0.19.9}[32462]: initializing ...

jvm 1 | [2012-09-18 16:49:03,403][INFO ][plugins ]
[ESNodeReader] loaded , sites [hadoop]

jvm 1 | [2012-09-18 16:49:03,932][ERROR][bootstrap ]
{0.19.9}: Initialization Failed ...

jvm 1 | - NoClassSettingsException[Failed to load class setting
[gateway.type] with value [hdfs]]

jvm 1 | ClassNotFoundException[hdfs]

So I tried to download and build the plugin to understand how it work and
how I can fix the _site configuration. I succeed in build it with Maeven,
but itch time I test my build I obtain the same error than before:

Running test.java.org.elasticsearch.hadoop.gateway.HdfsGatewayTests

Sep 18, 2012 4:38:10 PM org.elasticsearch.node

INFO: [node1] {0.19.0}[29968]: initializing ...

Sep 18, 2012 4:38:10 PM org.elasticsearch.plugins

INFO: [node1] loaded [hadoop], sites

Tests run: 5, Failures: 2, Errors: 0, Skipped: 3, Time elapsed: 0.783 sec
<<< FAILURE!

Results:

Failed tests:
setUpNodes(test.java.org.elasticsearch.hadoop.gateway.HdfsGatewayTests):
Failed to load class setting [gateway.type] with value [hdfs]

setUpNodes(test.java.org.elasticsearch.hadoop.gateway.HdfsGatewayTests):
Failed to load class setting [gateway.type] with value [hdfs]

Tests run: 5, Failures: 2, Errors: 0, Skipped: 3

Is there another way to build the plugin or a way to fix the Failed to
load hdfs class? I tried to reference the library instated of just hdfs but
I have the same result.

Thanks for your helps.

--

--

XCouly · September 19, 2012, 8:07am

On Tuesday, September 18, 2012 9:52:06 PM UTC+2, kimchy wrote:

How do you install the hadoop plugin? Btw, the shared gateway
configuration is not recommended and the local gateway is the recommended
gateway to use (its the default). See the previous mails here on what needs
to be done with the local gateway to improve the "backup / restore" aspect
to it.

For the Installation I use:

bin/plugin -install elasticsearch/elasticsearch-hadoop/1.2.0

And it returns me this kind of message:

-> Installing elasticsearch/elasticsearch-hadoop/1.2.0...

Trying
https://github.com/downloads/elasticsearch/elasticsearch-hadoop/elasticsearch-hadoop-1.2.0.zip...

Trying
https://github.com/elasticsearch/elasticsearch-hadoop/zipball/v1.2.0...

Downloading .......DONE

Identified as a _site plugin, moving to _site structure ...

Installed hadoop

For the use of shared gateway:

I understand that local gateway will do the same job as shared gateway,
thanks to remind mi to compare it with hadoop gateway. But here is my
question why is shared gateway not recommended? For my shared gateway allow
you to have a secure save external to ES cluster. So what will happen if
all my ES cluster crash can I be able to restore my entire Index just with
local gateway?

On Wednesday, September 19, 2012 12:44:28 AM UTC+2, Jörg Prante wrote:

Hadoop was designed for large datasets (TB to PB) with rock solid
reliability. You can use Hadoop as a long term storage for Elasticsearch
indexes and metadata (via the gateway mechanism).

Another method for using Hadoop is loading data with map/reduce into
Elasticsearch, for example see
http://thedatachef.blogspot.de/2011/01/bulk-indexing-with-elasticsearch-and.html
or
You know, for search. Querying 24 Billion Documents in 900ms | PPT

Jörg

Thanks those link, I will try it for using hadoop it will help me surely
for the next step of my test.

--

kimchy · September 28, 2012, 11:24am

can you check if you can wget / download: https://github.com/downloads/elasticsearch/elasticsearch-hadoop/elasticsearch-hadoop-1.2.0.zip ?

On Sep 19, 2012, at 10:07 AM, XCouly xavier.couly@gmail.com wrote:

On Tuesday, September 18, 2012 9:52:06 PM UTC+2, kimchy wrote:
How do you install the hadoop plugin? Btw, the shared gateway configuration is not recommended and the local gateway is the recommended gateway to use (its the default). See the previous mails here on what needs to be done with the local gateway to improve the "backup / restore" aspect to it.

For the Installation I use:

bin/plugin -install elasticsearch/elasticsearch-hadoop/1.2.0

And it returns me this kind of message:

-> Installing elasticsearch/elasticsearch-hadoop/1.2.0...
Trying https://github.com/downloads/elasticsearch/elasticsearch-hadoop/elasticsearch-hadoop-1.2.0.zip...
Trying https://github.com/elasticsearch/elasticsearch-hadoop/zipball/v1.2.0...
Downloading .......DONE
Identified as a _site plugin, moving to _site structure ...
Installed hadoop

For the use of shared gateway:

I understand that local gateway will do the same job as shared gateway, thanks to remind mi to compare it with hadoop gateway. But here is my question why is shared gateway not recommended? For my shared gateway allow you to have a secure save external to ES cluster. So what will happen if all my ES cluster crash can I be able to restore my entire Index just with local gateway?

On Wednesday, September 19, 2012 12:44:28 AM UTC+2, Jörg Prante wrote:
Hadoop was designed for large datasets (TB to PB) with rock solid reliability. You can use Hadoop as a long term storage for Elasticsearch indexes and metadata (via the gateway mechanism).

Another method for using Hadoop is loading data with map/reduce into Elasticsearch, for example see http://thedatachef.blogspot.de/2011/01/bulk-indexing-with-elasticsearch-and.html or You know, for search. Querying 24 Billion Documents in 900ms | PPT

Jörg

Thanks those link, I will try it for using hadoop it will help me surely for the next step of my test.

--

--