Problems getting hdfs gateway configured/started

Elastic Stack Elasticsearch
1

Hey all,

I've been trying to get ES running on top of hdfs and I've run into a
few problems.

I've got a completely vanilla installation of hadoop 0.20.203.0 which
I've confirmed is functioning by creating a directory "elastic",
copying a file to the directory, cat'ing the contents of the file and
then rm'ing the file.

I've also installed elasticsearch-0.17.8 and have installed the hadoop
plugin by running "bin/plugin -install hadoop"

At first I was getting an error in my hdfs logs like:

2011-10-11 15:56:41,942 WARN org.apache.hadoop.ipc.Server: Incorrect header or version mismatch from 127.0.0.1:59238 got version 3 expected version 4

which I resolved by copying the hadoop-core-0.20.203.0.jar from my
hadoop installation into the /plugins/hadoop directory and removing
the downrev, hadoop-core-0.20.2.jar, to /tmp per another thread on the
mailinglist archive.

I've now got the problem where starting elasticsearch fails with a
trace like:

$ bin/elasticsearch -f
[2011-10-11 16:01:59,571][INFO ][node ] [Neurotap]
{elasticsearch/0.17.8}[25408]: initializing ...
[2011-10-11 16:01:59,581][INFO ][plugins ] [Neurotap]
loaded [hadoop], sites
[2011-10-11 16:02:02,272][ERROR][bootstrap ]
{elasticsearch/0.17.8}: Initialization Failed ...

  1. NoClassDefFoundError[org/apache/commons/configuration/
    Configuration]

ClassNotFoundException[org.apache.commons.configuration.Configuration]2)
IOException[failure to login]
LoginException[java.lang.NoClassDefFoundError: Could not initialize
class org.apache.hadoop.security.KerberosName

Being a complete java idiot, and having zero experience with either
hdfs nor elasticsearch I'm completely at a loss as to how to fix this
or if this is a configuration error or some other type of error. Any
help would greatly appreciated.

-Dave

2

Quickie Update, looks like everything works just fine if I run hadoop
0.20.2, I guess that the hadoop plugin is just incompatible with the
newer hadoop api.

-Dave

On Oct 11, 4:01 pm, Dave Rawks d...@pandora.com wrote:

Hey all,

I've been trying to get ES running on top of hdfs and I've run into a
few problems.

I've got a completely vanilla installation of hadoop 0.20.203.0 which
I've confirmed is functioning by creating a directory "elastic",
copying a file to the directory, cat'ing the contents of the file and
then rm'ing the file.

I've also installed elasticsearch-0.17.8 and have installed the hadoop
plugin by running "bin/plugin -install hadoop"

At first I was getting an error in my hdfs logs like:

2011-10-11 15:56:41,942 WARN org.apache.hadoop.ipc.Server: Incorrect header or version mismatch from 127.0.0.1:59238 got version 3 expected version 4

which I resolved by copying the hadoop-core-0.20.203.0.jar from my
hadoop installation into the /plugins/hadoop directory and removing
the downrev, hadoop-core-0.20.2.jar, to /tmp per another thread on the
mailinglist archive.

I've now got the problem where starting elasticsearch fails with a
trace like:

$ bin/elasticsearch -f
[2011-10-11 16:01:59,571][INFO ][node ] [Neurotap]
{elasticsearch/0.17.8}[25408]: initializing ...
[2011-10-11 16:01:59,581][INFO ][plugins ] [Neurotap]
loaded [hadoop], sites
[2011-10-11 16:02:02,272][ERROR][bootstrap ]
{elasticsearch/0.17.8}: Initialization Failed ...

  1. NoClassDefFoundError[org/apache/commons/configuration/
    Configuration]

ClassNotFoundException[org.apache.commons.configuration.Configuration]2)
IOException[failure to login]
LoginException[java.lang.NoClassDefFoundError: Could not initialize
class org.apache.hadoop.security.KerberosName

Being a complete java idiot, and having zero experience with either
hdfs nor elasticsearch I'm completely at a loss as to how to fix this
or if this is a configuration error or some other type of error. Any
help would greatly appreciated.

-Dave

3

That might be the case, you can replace the hadoop jar in the hadoop plugin
with a newer version. Why not use the local gateway? I recommend it as it
has lower overhead.

On Wed, Oct 12, 2011 at 1:27 AM, Dave Rawks dave@pandora.com wrote:

Quickie Update, looks like everything works just fine if I run hadoop
0.20.2, I guess that the hadoop plugin is just incompatible with the
newer hadoop api.

-Dave

On Oct 11, 4:01 pm, Dave Rawks d...@pandora.com wrote:

Hey all,

I've been trying to get ES running on top of hdfs and I've run into a
few problems.

I've got a completely vanilla installation of hadoop 0.20.203.0 which
I've confirmed is functioning by creating a directory "elastic",
copying a file to the directory, cat'ing the contents of the file and
then rm'ing the file.

I've also installed elasticsearch-0.17.8 and have installed the hadoop
plugin by running "bin/plugin -install hadoop"

At first I was getting an error in my hdfs logs like:

2011-10-11 15:56:41,942 WARN org.apache.hadoop.ipc.Server: Incorrect
header or version mismatch from 127.0.0.1:59238 got version 3 expected
version 4

which I resolved by copying the hadoop-core-0.20.203.0.jar from my
hadoop installation into the /plugins/hadoop directory and removing
the downrev, hadoop-core-0.20.2.jar, to /tmp per another thread on the
mailinglist archive.

I've now got the problem where starting elasticsearch fails with a
trace like:

$ bin/elasticsearch -f
[2011-10-11 16:01:59,571][INFO ][node ] [Neurotap]
{elasticsearch/0.17.8}[25408]: initializing ...
[2011-10-11 16:01:59,581][INFO ][plugins ] [Neurotap]
loaded [hadoop], sites
[2011-10-11 16:02:02,272][ERROR][bootstrap ]
{elasticsearch/0.17.8}: Initialization Failed ...

  1. NoClassDefFoundError[org/apache/commons/configuration/
    Configuration]

ClassNotFoundException[org.apache.commons.configuration.Configuration]2)
IOException[failure to login]
LoginException[java.lang.NoClassDefFoundError: Could not
initialize
class org.apache.hadoop.security.KerberosName

Being a complete java idiot, and having zero experience with either
hdfs nor elasticsearch I'm completely at a loss as to how to fix this
or if this is a configuration error or some other type of error. Any
help would greatly appreciated.

-Dave