# ES does not start with Hadoop gateway
**URL:** https://discuss.elastic.co/t/es-does-not-start-with-hadoop-gateway/4858
**Category:** Elasticsearch
**Created:** [July 14, 2011, 1:04pm UTC](https://discuss.elastic.co/t/es-does-not-start-with-hadoop-gateway/4858 "2011-07-14T13:04:19Z")
**Posts on this page:** 5
**Page:** 1
### Author:  [@Uli\_Kohler](https://discuss.elastic.co/u/Uli_Kohler)
#### Post date: [July 14, 2011, 1:04pm UTC](https://discuss.elastic.co/t/es-does-not-start-with-hadoop-gateway/4858/1 "2011-07-14T13:04:19Z")
Hi,
I'm running ES 0.16.3 on an 21-node cluster and I want to use the Hadoop
gateway feature.
My gateway config looks like this:
gateway:
type: hdfs
hdfs:
uri: hdfs://ibis218-81:8020
path: elasticsearch
The URI seems to be correct:
uli@desktop:/home/uli$ hadoop fs -fs hdfs://ibis218-81:8020 -ls /
Found 8 items
drwxr-xr-x - uli supergroup 0 2011-07-07 16:54 /elasticsearch
drwxr-xr-x - hdfs supergroup 0 2011-07-05 11:35 /user
Anyway, ElasticSearch doesn't start up; the log only contains two lines:
[2011-07-14 14:53:23,696][INFO][node] [Pierce,
Alexander Goodwin] {elasticsearch/0.16.3}[375]: initializing ...
[2011-07-14 14:53:23,717][INFO][plugins] [Pierce,
Alexander Goodwin] loaded [river-twitter, hadoop, river-wikipedia]
After waiting more than 10 minutes, I stopped the ES server and restarted it
with the gateway section commented out - it worked perfectly.
Does anyone of you have an idea why ES doesn't start up?
Thanks in advance!
Best regards,
Uli
---
### Author:  [@kimchy](https://discuss.elastic.co/u/kimchy)
#### Post date: [July 15, 2011, 5:25am UTC](https://discuss.elastic.co/t/es-does-not-start-with-hadoop-gateway/4858/2 "2011-07-15T05:25:33Z")
Can you issue a thread dump when it happens? Which version of hadoop are you using? In general, its recommended to use the local gateway over shared gateway, btw.
On Thursday, July 14, 2011 at 4:04 PM, Uli Köhler wrote:
> Hi,
> I'm running ES 0.16.3 on an 21-node cluster and I want to use the Hadoop gateway feature.
>
> My gateway config looks like this:
>
> gateway:
> type: hdfs
> hdfs:
> uri: hdfs://ibis218-81:8020
> path: elasticsearch
>
> The URI seems to be correct:
> uli@desktop:/home/uli$ hadoop fs -fs hdfs://ibis218-81:8020 -ls /
> Found 8 items
> drwxr-xr-x - uli supergroup 0 2011-07-07 16:54 /elasticsearch
> drwxr-xr-x - hdfs supergroup 0 2011-07-05 11:35 /user
>
> Anyway, Elasticsearch doesn't start up; the log only contains two lines:
>
> [2011-07-14 14:53:23,696][INFO][node] [Pierce, Alexander Goodwin] {elasticsearch/0.16.3}[375]: initializing ...
> [2011-07-14 14:53:23,717][INFO][plugins] [Pierce, Alexander Goodwin] loaded [river-twitter, hadoop, river-wikipedia]
>
> After waiting more than 10 minutes, I stopped the ES server and restarted it with the gateway section commented out - it worked perfectly.
>
> Does anyone of you have an idea why ES doesn't start up?
>
> Thanks in advance!
>
> Best regards,
> Uli
---
### Author:  [@Uli\_Kohler](https://discuss.elastic.co/u/Uli_Kohler)
#### Post date: [July 15, 2011, 12:19pm UTC](https://discuss.elastic.co/t/es-does-not-start-with-hadoop-gateway/4858/3 "2011-07-15T12:19:30Z")
Hi Shay,
thanks for your quick answer!
I reproduced the problem (with 0.16.3) by starting ES with gateway enabled
on my computer (only one node) by executing
sudo bin/elasticsearch -f
It fails after a few seconds:
[2011-07-15 14:05:41,424][INFO][node] [Mastermind]
{elasticsearch/0.16.3}[1578]: initializing ...
[2011-07-15 14:05:41,429][INFO][plugins] [Mastermind]
loaded [hadoop]
[2011-07-15 14:05:43,791][ERROR][bootstrap]
{elasticsearch/0.16.3}: Initialization Failed ...
1. IOException[Call to ibis218-81/146.107.218.81:8020 failed on local
exception: java.io.EOFException]
We're currently usings Cloudera's distribution for hadoop in version CDH3U0.
Basically this is hadoop 0.20.2.
I'm trying to use hadoop gateway because I think it provides better failsafe
mechanisms for my usecase. If the HDD of one node dies, do I get problems
with the local gateway?
Best regards and many thanks,
Uli
2011/7/15 Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)
> Can you issue a thread dump when it happens? Which version of hadoop are
> you using? In general, its recommended to use the local gateway over shared
> gateway, btw.
>
> On Thursday, July 14, 2011 at 4:04 PM, Uli Köhler wrote:
>
> Hi,
> I'm running ES 0.16.3 on an 21-node cluster and I want to use the Hadoop
> gateway feature.
>
> My gateway config looks like this:
>
> gateway:
> type: hdfs
> hdfs:
> uri: hdfs://ibis218-81:8020
> path: elasticsearch
>
> The URI seems to be correct:
> uli@desktop:/home/uli$ hadoop fs -fs hdfs://ibis218-81:8020 -ls /
> Found 8 items
> drwxr-xr-x - uli supergroup 0 2011-07-07 16:54 /elasticsearch
> drwxr-xr-x - hdfs supergroup 0 2011-07-05 11:35 /user
>
> Anyway, Elasticsearch doesn't start up; the log only contains two lines:
>
> [2011-07-14 14:53:23,696][INFO][node] [Pierce,
> Alexander Goodwin] {elasticsearch/0.16.3}[375]: initializing ...
> [2011-07-14 14:53:23,717][INFO][plugins] [Pierce,
> Alexander Goodwin] loaded [river-twitter, hadoop, river-wikipedia]
>
> After waiting more than 10 minutes, I stopped the ES server and restarted
> it with the gateway section commented out - it worked perfectly.
>
> Does anyone of you have an idea why ES doesn't start up?
>
> Thanks in advance!
>
> Best regards,
> Uli
---
### Author:  [@kimchy](https://discuss.elastic.co/u/kimchy)
#### Post date: [July 15, 2011, 10:00pm UTC](https://discuss.elastic.co/t/es-does-not-start-with-hadoop-gateway/4858/4 "2011-07-15T22:00:04Z")
If the HDD on one node fails when using local gateway, the replicas will be used (and the failed shards will be allocated to the rest of the cluster).
Maybe you can set org.hadoop logging level higher and check why it fails? Also, maybe replace the plugin hadoop lib with the cloudera distribution jar file?
On Friday, July 15, 2011 at 3:19 PM, Uli Köhler wrote:
> Hi Shay,
> thanks for your quick answer!
>
> I reproduced the problem (with 0.16.3) by starting ES with gateway enabled on my computer (only one node) by executing
> sudo bin/elasticsearch -f
>
> It fails after a few seconds:
>
> [2011-07-15 14:05:41,424][INFO][node] [Mastermind] {elasticsearch/0.16.3}[1578]: initializing ...
> [2011-07-15 14:05:41,429][INFO][plugins] [Mastermind] loaded [hadoop]
> [2011-07-15 14:05:43,791][ERROR][bootstrap] {elasticsearch/0.16.3}: Initialization Failed ...
>
> 1. IOException[Call to ibis218-81/146.107.218.81:8020 ([http://146.107.218.81:8020](http://146.107.218.81:8020)) failed on local exception: java.io.EOFException]
>
> We're currently usings Cloudera's distribution for hadoop in version CDH3U0. Basically this is hadoop 0.20.2.
>
> I'm trying to use hadoop gateway because I think it provides better failsafe mechanisms for my usecase. If the HDD of one node dies, do I get problems with the local gateway?
>
> Best regards and many thanks,
> Uli
>
> 2011/7/15 Shay Banon \<[shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) ([mailto:shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com))\>
>
> > Can you issue a thread dump when it happens? Which version of hadoop are you using? In general, its recommended to use the local gateway over shared gateway, btw.
> >
> > On Thursday, July 14, 2011 at 4:04 PM, Uli Köhler wrote:
> >
> > > Hi,
> > > I'm running ES 0.16.3 on an 21-node cluster and I want to use the Hadoop gateway feature.
> > >
> > > My gateway config looks like this:
> > >
> > > gateway:
> > > type: hdfs
> > > hdfs:
> > > uri: hdfs://ibis218-81:8020
> > > path: elasticsearch
> > >
> > > The URI seems to be correct:
> > > uli@desktop:/home/uli$ hadoop fs -fs hdfs://ibis218-81:8020 -ls /
> > > Found 8 items
> > > drwxr-xr-x - uli supergroup 0 2011-07-07 16:54 /elasticsearch
> > > drwxr-xr-x - hdfs supergroup 0 2011-07-05 11:35 /user
> > >
> > > Anyway, Elasticsearch doesn't start up; the log only contains two lines:
> > >
> > > [2011-07-14 14:53:23,696][INFO][node] [Pierce, Alexander Goodwin] {elasticsearch/0.16.3}[375]: initializing ...
> > > [2011-07-14 14:53:23,717][INFO][plugins] [Pierce, Alexander Goodwin] loaded [river-twitter, hadoop, river-wikipedia]
> > >
> > > After waiting more than 10 minutes, I stopped the ES server and restarted it with the gateway section commented out - it worked perfectly.
> > >
> > > Does anyone of you have an idea why ES doesn't start up?
> > >
> > > Thanks in advance!
> > >
> > > Best regards,
> > > Uli
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 6, 2017, 4:00am UTC](https://discuss.elastic.co/t/es-does-not-start-with-hadoop-gateway/4858/5 "2017-07-06T04:00:29Z")