ES does not start with Hadoop gateway

Hi,
I'm running ES 0.16.3 on an 21-node cluster and I want to use the Hadoop
gateway feature.

My gateway config looks like this:

gateway:
type: hdfs
hdfs:
uri: hdfs://ibis218-81:8020
path: elasticsearch

The URI seems to be correct:
uli@desktop:/home/uli$ hadoop fs -fs hdfs://ibis218-81:8020 -ls /
Found 8 items
drwxr-xr-x - uli supergroup 0 2011-07-07 16:54 /elasticsearch
drwxr-xr-x - hdfs supergroup 0 2011-07-05 11:35 /user

Anyway, ElasticSearch doesn't start up; the log only contains two lines:

[2011-07-14 14:53:23,696][INFO ][node ] [Pierce,
Alexander Goodwin] {elasticsearch/0.16.3}[375]: initializing ...
[2011-07-14 14:53:23,717][INFO ][plugins ] [Pierce,
Alexander Goodwin] loaded [river-twitter, hadoop, river-wikipedia]

After waiting more than 10 minutes, I stopped the ES server and restarted it
with the gateway section commented out - it worked perfectly.

Does anyone of you have an idea why ES doesn't start up?

Thanks in advance!

Best regards,
Uli

Can you issue a thread dump when it happens? Which version of hadoop are you using? In general, its recommended to use the local gateway over shared gateway, btw.

On Thursday, July 14, 2011 at 4:04 PM, Uli Köhler wrote:

Hi,
I'm running ES 0.16.3 on an 21-node cluster and I want to use the Hadoop gateway feature.

My gateway config looks like this:

gateway:
type: hdfs
hdfs:
uri: hdfs://ibis218-81:8020
path: elasticsearch

The URI seems to be correct:
uli@desktop:/home/uli$ hadoop fs -fs hdfs://ibis218-81:8020 -ls /
Found 8 items
drwxr-xr-x - uli supergroup 0 2011-07-07 16:54 /elasticsearch
drwxr-xr-x - hdfs supergroup 0 2011-07-05 11:35 /user

Anyway, Elasticsearch doesn't start up; the log only contains two lines:

[2011-07-14 14:53:23,696][INFO ][node ] [Pierce, Alexander Goodwin] {elasticsearch/0.16.3}[375]: initializing ...
[2011-07-14 14:53:23,717][INFO ][plugins ] [Pierce, Alexander Goodwin] loaded [river-twitter, hadoop, river-wikipedia]

After waiting more than 10 minutes, I stopped the ES server and restarted it with the gateway section commented out - it worked perfectly.

Does anyone of you have an idea why ES doesn't start up?

Thanks in advance!

Best regards,
Uli

Hi Shay,
thanks for your quick answer!

I reproduced the problem (with 0.16.3) by starting ES with gateway enabled
on my computer (only one node) by executing
sudo bin/elasticsearch -f

It fails after a few seconds:

[2011-07-15 14:05:41,424][INFO ][node ] [Mastermind]
{elasticsearch/0.16.3}[1578]: initializing ...
[2011-07-15 14:05:41,429][INFO ][plugins ] [Mastermind]
loaded [hadoop]
[2011-07-15 14:05:43,791][ERROR][bootstrap ]
{elasticsearch/0.16.3}: Initialization Failed ...

  1. IOException[Call to ibis218-81/146.107.218.81:8020 failed on local
    exception: java.io.EOFException]

We're currently usings Cloudera's distribution for hadoop in version CDH3U0.
Basically this is hadoop 0.20.2.

I'm trying to use hadoop gateway because I think it provides better failsafe
mechanisms for my usecase. If the HDD of one node dies, do I get problems
with the local gateway?

Best regards and many thanks,
Uli

2011/7/15 Shay Banon shay.banon@elasticsearch.com

Can you issue a thread dump when it happens? Which version of hadoop are
you using? In general, its recommended to use the local gateway over shared
gateway, btw.

On Thursday, July 14, 2011 at 4:04 PM, Uli Köhler wrote:

Hi,
I'm running ES 0.16.3 on an 21-node cluster and I want to use the Hadoop
gateway feature.

My gateway config looks like this:

gateway:
type: hdfs
hdfs:
uri: hdfs://ibis218-81:8020
path: elasticsearch

The URI seems to be correct:
uli@desktop:/home/uli$ hadoop fs -fs hdfs://ibis218-81:8020 -ls /
Found 8 items
drwxr-xr-x - uli supergroup 0 2011-07-07 16:54 /elasticsearch
drwxr-xr-x - hdfs supergroup 0 2011-07-05 11:35 /user

Anyway, Elasticsearch doesn't start up; the log only contains two lines:

[2011-07-14 14:53:23,696][INFO ][node ] [Pierce,
Alexander Goodwin] {elasticsearch/0.16.3}[375]: initializing ...
[2011-07-14 14:53:23,717][INFO ][plugins ] [Pierce,
Alexander Goodwin] loaded [river-twitter, hadoop, river-wikipedia]

After waiting more than 10 minutes, I stopped the ES server and restarted
it with the gateway section commented out - it worked perfectly.

Does anyone of you have an idea why ES doesn't start up?

Thanks in advance!

Best regards,
Uli

If the HDD on one node fails when using local gateway, the replicas will be used (and the failed shards will be allocated to the rest of the cluster).

Maybe you can set org.hadoop logging level higher and check why it fails? Also, maybe replace the plugin hadoop lib with the cloudera distribution jar file?

On Friday, July 15, 2011 at 3:19 PM, Uli Köhler wrote:

Hi Shay,
thanks for your quick answer!

I reproduced the problem (with 0.16.3) by starting ES with gateway enabled on my computer (only one node) by executing
sudo bin/elasticsearch -f

It fails after a few seconds:

[2011-07-15 14:05:41,424][INFO ][node ] [Mastermind] {elasticsearch/0.16.3}[1578]: initializing ...
[2011-07-15 14:05:41,429][INFO ][plugins ] [Mastermind] loaded [hadoop]
[2011-07-15 14:05:43,791][ERROR][bootstrap ] {elasticsearch/0.16.3}: Initialization Failed ...

  1. IOException[Call to ibis218-81/146.107.218.81:8020 (http://146.107.218.81:8020) failed on local exception: java.io.EOFException]

We're currently usings Cloudera's distribution for hadoop in version CDH3U0. Basically this is hadoop 0.20.2.

I'm trying to use hadoop gateway because I think it provides better failsafe mechanisms for my usecase. If the HDD of one node dies, do I get problems with the local gateway?

Best regards and many thanks,
Uli

2011/7/15 Shay Banon <shay.banon@elasticsearch.com (mailto:shay.banon@elasticsearch.com)>

Can you issue a thread dump when it happens? Which version of hadoop are you using? In general, its recommended to use the local gateway over shared gateway, btw.

On Thursday, July 14, 2011 at 4:04 PM, Uli Köhler wrote:

Hi,
I'm running ES 0.16.3 on an 21-node cluster and I want to use the Hadoop gateway feature.

My gateway config looks like this:

gateway:
type: hdfs
hdfs:
uri: hdfs://ibis218-81:8020
path: elasticsearch

The URI seems to be correct:
uli@desktop:/home/uli$ hadoop fs -fs hdfs://ibis218-81:8020 -ls /
Found 8 items
drwxr-xr-x - uli supergroup 0 2011-07-07 16:54 /elasticsearch
drwxr-xr-x - hdfs supergroup 0 2011-07-05 11:35 /user

Anyway, Elasticsearch doesn't start up; the log only contains two lines:

[2011-07-14 14:53:23,696][INFO ][node ] [Pierce, Alexander Goodwin] {elasticsearch/0.16.3}[375]: initializing ...
[2011-07-14 14:53:23,717][INFO ][plugins ] [Pierce, Alexander Goodwin] loaded [river-twitter, hadoop, river-wikipedia]

After waiting more than 10 minutes, I stopped the ES server and restarted it with the gateway section commented out - it worked perfectly.

Does anyone of you have an idea why ES doesn't start up?

Thanks in advance!

Best regards,
Uli