# Problems with Hadoop plugin
**URL:** https://discuss.elastic.co/t/problems-with-hadoop-plugin/9051
**Category:** Elasticsearch
**Created:** [September 18, 2012, 2:59pm UTC](https://discuss.elastic.co/t/problems-with-hadoop-plugin/9051 "2012-09-18T14:59:21Z")
**Posts on this page:** 7
**Page:** 1
### Author:  [@XCouly](https://discuss.elastic.co/u/XCouly)
#### Post date: [September 18, 2012, 2:59pm UTC](https://discuss.elastic.co/t/problems-with-hadoop-plugin/9051/1 "2012-09-18T14:59:21Z")
I tried to use a Hadoop Gateway on ES 19.9. Every time I tried to download
Hadoop Gateway it's been installing in \_site configuration.
When I tried to configure Hadoop gateway with this kind of configuration in
my elasticsearch.yml:
gateway.type : hdfs
gateway.hdfs.uri : hdfs://:8020
gateway.hdfs.path : /user/elasticsearch
it result on an error like this:
jvm 1 | [2012-09-18 16:49:03,355][INFO][node]
[ESNodeReader] {0.19.9}[32462]: initializing ...
jvm 1 | [2012-09-18 16:49:03,403][INFO][plugins]
[ESNodeReader] loaded [], sites [hadoop]
jvm 1 | [2012-09-18 16:49:03,932][ERROR][bootstrap]
{0.19.9}: Initialization Failed ...
jvm 1 | - NoClassSettingsException[Failed to load class setting
[gateway.type] with value [hdfs]]
jvm 1 | ClassNotFoundException[hdfs]
So I tried to download and build the plugin to understand how it work and
how I can fix the \_site configuration. I succeed in build it with Maeven,
but itch time I test my build I obtain the same error than before:
Running test.java.org.elasticsearch.hadoop.gateway.HdfsGatewayTests
Sep 18, 2012 4:38:10 PM org.elasticsearch.node
INFO: [node1] {0.19.0}[29968]: initializing ...
Sep 18, 2012 4:38:10 PM org.elasticsearch.plugins
INFO: [node1] loaded [hadoop], sites []
Tests run: 5, Failures: 2, Errors: 0, Skipped: 3, Time elapsed: 0.783 sec
\<\<\< FAILURE!
Results:
Failed tests:
setUpNodes(test.java.org.elasticsearch.hadoop.gateway.HdfsGatewayTests):
Failed to load class setting [gateway.type] with value [hdfs]
setUpNodes(test.java.org.elasticsearch.hadoop.gateway.HdfsGatewayTests):
Failed to load class setting [gateway.type] with value [hdfs]
Tests run: 5, Failures: 2, Errors: 0, Skipped: 3
Is there another way to build the plugin or a way to fix the Failed to load
hdfs class? I tried to reference the library instated of just hdfs but I
have the same result.
Thanks for your helps.
--
---
### Author:  [@kimchy](https://discuss.elastic.co/u/kimchy)
#### Post date: [September 18, 2012, 7:52pm UTC](https://discuss.elastic.co/t/problems-with-hadoop-plugin/9051/2 "2012-09-18T19:52:00Z")
How do you install the hadoop plugin? Btw, the shared gateway configuration is not recommended and the local gateway is the recommended gateway to use (its the default). See the previous mails here on what needs to be done with the local gateway to improve the "backup / restore" aspect to it.
On Sep 18, 2012, at 4:59 PM, XCouly [xavier.couly@gmail.com](mailto:xavier.couly@gmail.com) wrote:
> I tried to use a Hadoop Gateway on ES 19.9. Every time I tried to download Hadoop Gateway it's been installing in \_site configuration.
>
> When I tried to configure Hadoop gateway with this kind of configuration in my elasticsearch.yml:
>
> gateway.type : hdfs
> gateway.hdfs.uri : hdfs://:8020
> gateway.hdfs.path : /user/elasticsearch
>
> it result on an error like this:
>
> jvm 1 | [2012-09-18 16:49:03,355][INFO][node] [ESNodeReader] {0.19.9}[32462]: initializing ...
> jvm 1 | [2012-09-18 16:49:03,403][INFO][plugins] [ESNodeReader] loaded , sites [hadoop]
> jvm 1 | [2012-09-18 16:49:03,932][ERROR][bootstrap] {0.19.9}: Initialization Failed ...
> jvm 1 | - NoClassSettingsException[Failed to load class setting [gateway.type] with value [hdfs]]
> jvm 1 | ClassNotFoundException[hdfs]
>
> So I tried to download and build the plugin to understand how it work and how I can fix the \_site configuration. I succeed in build it with Maeven, but itch time I test my build I obtain the same error than before:
>
> Running test.java.org.elasticsearch.hadoop.gateway.HdfsGatewayTests
> Sep 18, 2012 4:38:10 PM org.elasticsearch.node
> INFO: [node1] {0.19.0}[29968]: initializing ...
> Sep 18, 2012 4:38:10 PM org.elasticsearch.plugins
> INFO: [node1] loaded [hadoop], sites
> Tests run: 5, Failures: 2, Errors: 0, Skipped: 3, Time elapsed: 0.783 sec \<\<\< FAILURE!
>
> Results:
>
> Failed tests: setUpNodes(test.java.org.elasticsearch.hadoop.gateway.HdfsGatewayTests): Failed to load class setting [gateway.type] with value [hdfs]
> setUpNodes(test.java.org.elasticsearch.hadoop.gateway.HdfsGatewayTests): Failed to load class setting [gateway.type] with value [hdfs]
>
> Tests run: 5, Failures: 2, Errors: 0, Skipped: 3
>
> Is there another way to build the plugin or a way to fix the Failed to load hdfs class? I tried to reference the library instated of just hdfs but I have the same result.
>
> Thanks for your helps.
>
> --
--
---
### Author:  [@WhatElse](https://discuss.elastic.co/u/WhatElse)
#### Post date: [September 18, 2012, 8:21pm UTC](https://discuss.elastic.co/t/problems-with-hadoop-plugin/9051/3 "2012-09-18T20:21:08Z")
I don't know if I'm wrong or not but which advantage brings Hadoop for
Elasticsearch ?
Le mardi 18 septembre 2012 15:52:06 UTC-4, kimchy a écrit :
> How do you install the hadoop plugin? Btw, the shared gateway
> configuration is not recommended and the local gateway is the recommended
> gateway to use (its the default). See the previous mails here on what needs
> to be done with the local gateway to improve the "backup / restore" aspect
> to it.
>
> On Sep 18, 2012, at 4:59 PM, XCouly \<[xavier...@gmail.com](mailto:xavier...@gmail.com) \\>
> wrote:
>
> I tried to use a Hadoop Gateway on ES 19.9. Every time I tried to download
> Hadoop Gateway it's been installing in \_site configuration.
>
> When I tried to configure Hadoop gateway with this kind of configuration
> in my elasticsearch.yml:
>
> gateway.type : hdfs
>
> gateway.hdfs.uri : hdfs://:8020
>
> gateway.hdfs.path : /user/elasticsearch
>
> it result on an error like this:
>
> jvm 1 | [2012-09-18 16:49:03,355][INFO][node]
> [ESNodeReader] {0.19.9}[32462]: initializing ...
>
> jvm 1 | [2012-09-18 16:49:03,403][INFO][plugins]
> [ESNodeReader] loaded , sites [hadoop]
>
> jvm 1 | [2012-09-18 16:49:03,932][ERROR][bootstrap]
> {0.19.9}: Initialization Failed ...
>
> jvm 1 | - NoClassSettingsException[Failed to load class setting
> [gateway.type] with value [hdfs]]
>
> jvm 1 | ClassNotFoundException[hdfs]
>
> So I tried to download and build the plugin to understand how it work and
> how I can fix the \_site configuration. I succeed in build it with Maeven,
> but itch time I test my build I obtain the same error than before:
>
> Running test.java.org.elasticsearch.hadoop.gateway.HdfsGatewayTests
>
> Sep 18, 2012 4:38:10 PM org.elasticsearch.node
>
> INFO: [node1] {0.19.0}[29968]: initializing ...
>
> Sep 18, 2012 4:38:10 PM org.elasticsearch.plugins
>
> INFO: [node1] loaded [hadoop], sites
>
> Tests run: 5, Failures: 2, Errors: 0, Skipped: 3, Time elapsed: 0.783 sec
> \<\<\< FAILURE!
>
> Results:
>
> Failed tests:
> setUpNodes(test.java.org.elasticsearch.hadoop.gateway.HdfsGatewayTests):
> Failed to load class setting [gateway.type] with value [hdfs]
>
> setUpNodes(test.java.org.elasticsearch.hadoop.gateway.HdfsGatewayTests):
> Failed to load class setting [gateway.type] with value [hdfs]
>
> Tests run: 5, Failures: 2, Errors: 0, Skipped: 3
>
> Is there another way to build the plugin or a way to fix the Failed to
> load hdfs class? I tried to reference the library instated of just hdfs but
> I have the same result.
>
> Thanks for your helps.
>
> --
--
---
### Author:  [@jprante](https://discuss.elastic.co/u/jprante)
#### Post date: [September 18, 2012, 10:44pm UTC](https://discuss.elastic.co/t/problems-with-hadoop-plugin/9051/4 "2012-09-18T22:44:28Z")
Hadoop was designed for large datasets (TB to PB) with rock solid
reliability. You can use Hadoop as a long term storage for Elasticsearch
indexes and metadata (via the gateway mechanism).
Another method for using Hadoop is loading data with map/reduce into
Elasticsearch, for example
see [http://thedatachef.blogspot.de/2011/01/bulk-indexing-with-elasticsearch-and.html](http://thedatachef.blogspot.de/2011/01/bulk-indexing-with-elasticsearch-and.html)
or [You know, for search. Querying 24 Billion Documents in 900ms | PPT](http://de.slideshare.net/jodok1/you-know-for-search-querying-24-billion-documents-in-900ms)
Jörg
On Tuesday, September 18, 2012 10:21:08 PM UTC+2, WhatElse wrote:
> I don't know if I'm wrong or not but which advantage brings Hadoop for
> Elasticsearch ?
>
> Le mardi 18 septembre 2012 15:52:06 UTC-4, kimchy a écrit :
>
> > How do you install the hadoop plugin? Btw, the shared gateway
> > configuration is not recommended and the local gateway is the recommended
> > gateway to use (its the default). See the previous mails here on what needs
> > to be done with the local gateway to improve the "backup / restore" aspect
> > to it.
> >
> > On Sep 18, 2012, at 4:59 PM, XCouly [xavier...@gmail.com](mailto:xavier...@gmail.com) wrote:
> >
> > I tried to use a Hadoop Gateway on ES 19.9. Every time I tried to
> > download Hadoop Gateway it's been installing in \_site configuration.
> >
> > When I tried to configure Hadoop gateway with this kind of configuration
> > in my elasticsearch.yml:
> >
> > gateway.type : hdfs
> >
> > gateway.hdfs.uri : hdfs://:8020
> >
> > gateway.hdfs.path : /user/elasticsearch
> >
> > it result on an error like this:
> >
> > jvm 1 | [2012-09-18 16:49:03,355][INFO][node]
> > [ESNodeReader] {0.19.9}[32462]: initializing ...
> >
> > jvm 1 | [2012-09-18 16:49:03,403][INFO][plugins]
> > [ESNodeReader] loaded , sites [hadoop]
> >
> > jvm 1 | [2012-09-18 16:49:03,932][ERROR][bootstrap]
> > {0.19.9}: Initialization Failed ...
> >
> > jvm 1 | - NoClassSettingsException[Failed to load class setting
> > [gateway.type] with value [hdfs]]
> >
> > jvm 1 | ClassNotFoundException[hdfs]
> >
> > So I tried to download and build the plugin to understand how it work and
> > how I can fix the \_site configuration. I succeed in build it with Maeven,
> > but itch time I test my build I obtain the same error than before:
> >
> > Running test.java.org.elasticsearch.hadoop.gateway.HdfsGatewayTests
> >
> > Sep 18, 2012 4:38:10 PM org.elasticsearch.node
> >
> > INFO: [node1] {0.19.0}[29968]: initializing ...
> >
> > Sep 18, 2012 4:38:10 PM org.elasticsearch.plugins
> >
> > INFO: [node1] loaded [hadoop], sites
> >
> > Tests run: 5, Failures: 2, Errors: 0, Skipped: 3, Time elapsed: 0.783 sec
> > \<\<\< FAILURE!
> >
> > Results:
> >
> > Failed tests:
> > setUpNodes(test.java.org.elasticsearch.hadoop.gateway.HdfsGatewayTests):
> > Failed to load class setting [gateway.type] with value [hdfs]
> >
> > setUpNodes(test.java.org.elasticsearch.hadoop.gateway.HdfsGatewayTests):
> > Failed to load class setting [gateway.type] with value [hdfs]
> >
> > Tests run: 5, Failures: 2, Errors: 0, Skipped: 3
> >
> > Is there another way to build the plugin or a way to fix the Failed to
> > load hdfs class? I tried to reference the library instated of just hdfs but
> > I have the same result.
> >
> > Thanks for your helps.
> >
> > --
--
---
### Author:  [@XCouly](https://discuss.elastic.co/u/XCouly)
#### Post date: [September 19, 2012, 8:07am UTC](https://discuss.elastic.co/t/problems-with-hadoop-plugin/9051/5 "2012-09-19T08:07:57Z")
On Tuesday, September 18, 2012 9:52:06 PM UTC+2, kimchy wrote:
> How do you install the hadoop plugin? Btw, the shared gateway
> configuration is not recommended and the local gateway is the recommended
> gateway to use (its the default). See the previous mails here on what needs
> to be done with the local gateway to improve the "backup / restore" aspect
> to it.
For the Installation I use:
bin/plugin -install elasticsearch/elasticsearch-hadoop/1.2.0
And it returns me this kind of message:
-\> Installing elasticsearch/elasticsearch-hadoop/1.2.0...
Trying
[https://github.com/downloads/elasticsearch/elasticsearch-hadoop/elasticsearch-hadoop-1.2.0.zip](https://github.com/downloads/elasticsearch/elasticsearch-hadoop/elasticsearch-hadoop-1.2.0.zip)...
Trying
[https://github.com/elasticsearch/elasticsearch-hadoop/zipball/v1.2.0](https://github.com/elasticsearch/elasticsearch-hadoop/zipball/v1.2.0)...
Downloading .......DONE
Identified as a \_site plugin, moving to \_site structure ...
Installed hadoop
For the use of shared gateway:
I understand that local gateway will do the same job as shared gateway,
thanks to remind mi to compare it with hadoop gateway. But here is my
question why is shared gateway not recommended? For my shared gateway allow
you to have a secure save external to ES cluster. So what will happen if
all my ES cluster crash can I be able to restore my entire Index just with
local gateway?
On Wednesday, September 19, 2012 12:44:28 AM UTC+2, Jörg Prante wrote:
> Hadoop was designed for large datasets (TB to PB) with rock solid
> reliability. You can use Hadoop as a long term storage for Elasticsearch
> indexes and metadata (via the gateway mechanism).
>
> Another method for using Hadoop is loading data with map/reduce into
> Elasticsearch, for example see
> [http://thedatachef.blogspot.de/2011/01/bulk-indexing-with-elasticsearch-and.html](http://thedatachef.blogspot.de/2011/01/bulk-indexing-with-elasticsearch-and.html)
> or
> [You know, for search. Querying 24 Billion Documents in 900ms | PPT](http://de.slideshare.net/jodok1/you-know-for-search-querying-24-billion-documents-in-900ms)
>
> Jörg
Thanks those link, I will try it for using hadoop it will help me surely
for the next step of my test.
--
---
### Author:  [@kimchy](https://discuss.elastic.co/u/kimchy)
#### Post date: [September 28, 2012, 11:24am UTC](https://discuss.elastic.co/t/problems-with-hadoop-plugin/9051/6 "2012-09-28T11:24:47Z")
can you check if you can wget / download: [https://github.com/downloads/elasticsearch/elasticsearch-hadoop/elasticsearch-hadoop-1.2.0.zip](https://github.com/downloads/elasticsearch/elasticsearch-hadoop/elasticsearch-hadoop-1.2.0.zip) ?
On Sep 19, 2012, at 10:07 AM, XCouly [xavier.couly@gmail.com](mailto:xavier.couly@gmail.com) wrote:
> On Tuesday, September 18, 2012 9:52:06 PM UTC+2, kimchy wrote:
> How do you install the hadoop plugin? Btw, the shared gateway configuration is not recommended and the local gateway is the recommended gateway to use (its the default). See the previous mails here on what needs to be done with the local gateway to improve the "backup / restore" aspect to it.
>
> For the Installation I use:
>
> bin/plugin -install elasticsearch/elasticsearch-hadoop/1.2.0
>
> And it returns me this kind of message:
>
> -\> Installing elasticsearch/elasticsearch-hadoop/1.2.0...
> Trying [https://github.com/downloads/elasticsearch/elasticsearch-hadoop/elasticsearch-hadoop-1.2.0.zip](https://github.com/downloads/elasticsearch/elasticsearch-hadoop/elasticsearch-hadoop-1.2.0.zip)...
> Trying [https://github.com/elasticsearch/elasticsearch-hadoop/zipball/v1.2.0](https://github.com/elasticsearch/elasticsearch-hadoop/zipball/v1.2.0)...
> Downloading .......DONE
> Identified as a \_site plugin, moving to \_site structure ...
> Installed hadoop
>
> For the use of shared gateway:
>
> I understand that local gateway will do the same job as shared gateway, thanks to remind mi to compare it with hadoop gateway. But here is my question why is shared gateway not recommended? For my shared gateway allow you to have a secure save external to ES cluster. So what will happen if all my ES cluster crash can I be able to restore my entire Index just with local gateway?
>
> On Wednesday, September 19, 2012 12:44:28 AM UTC+2, Jörg Prante wrote:
> Hadoop was designed for large datasets (TB to PB) with rock solid reliability. You can use Hadoop as a long term storage for Elasticsearch indexes and metadata (via the gateway mechanism).
>
> Another method for using Hadoop is loading data with map/reduce into Elasticsearch, for example see [http://thedatachef.blogspot.de/2011/01/bulk-indexing-with-elasticsearch-and.html](http://thedatachef.blogspot.de/2011/01/bulk-indexing-with-elasticsearch-and.html) or [You know, for search. Querying 24 Billion Documents in 900ms | PPT](http://de.slideshare.net/jodok1/you-know-for-search-querying-24-billion-documents-in-900ms)
>
> Jörg
>
> Thanks those link, I will try it for using hadoop it will help me surely for the next step of my test.
>
> --
--
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 6, 2017, 3:10am UTC](https://discuss.elastic.co/t/problems-with-hadoop-plugin/9051/7 "2017-07-06T03:10:58Z")