I've followed the various guides to get going with the
elasticsearch-hadoop-integration in Hive, but I run into some issue:
add jar hdfs://host:9000//lib/elasticsearch-hadoop-hive-2.1.0.Beta4.jar;
INFO : converting to local hdfs:
//host:9000//lib/elasticsearch-hadoop-hive-2.1.0.Beta4.jar
INFO : Added [/tmp/15207d6b-e4b5-446b-bbe2-cff282056983_resources/
elasticsearch-hadoop-hive-2.1.0.Beta4.jar] to class path
INFO : Added resources: [hdfs:
//host:9000//lib/elasticsearch-hadoop-hive-2.1.0.Beta4.jar]
select * from estest;
Error: java.lang.NoClassDefFoundError: org/elasticsearch/hadoop/
serialization/dto/Node (state=,code=0)
As you can see I've followed the recommendation to put the jar file in
HDFS, and it seems like the jar is picked up in the classpath since without
the 'add jar' I get another error stating that the EsStorageHandler can't
be found. Any clues as to why this is happening?
It seems you are running into a classpath problem. The class mentioned in the exception
(org/elasticsearch/hadoop/serialization/dto/Node) is part of the elasticsearch-hadoop-hive-XXX. jar - you can verify
this yourself.
The fact that it is not found at runtime suggests that the a different or incomplete jar is used instead. This can occur
for example if a different jar is available in the Hive/Hadoop classpath which is picked up automatically and overrides
the one you use in your script.
So first try and double check the existing classpath - in the vast majority of Hive problems, this was the issue (and
old version was picked up instead). You can also verify this by trying to register the table - you should get an
exception right away. Once that's done, try different ways of adding the jar to your script classpath - it might be that
beeline has a different mechanism than vanilla Hive.
Hope this helps,
On 4/29/15 12:58 AM, Rasmus Aveskogh wrote:
Hi!
I've followed the various guides to get going with the elasticsearch-hadoop-integration in Hive, but I run into some
issue:
|
add jar hdfs://host:9000//lib/elasticsearch-hadoop-hive-2.1.0.Beta4.jar;
INFO :converting to localhdfs://host:9000//lib/elasticsearch-hadoop-hive-2.1.0.Beta4.jar
INFO :Added[/tmp/15207d6b-e4b5-446b-bbe2-cff282056983_resources/elasticsearch-hadoop-hive-2.1.0.Beta4.jar]to classpath
INFO :Addedresources:[hdfs://host:9000//lib/elasticsearch-hadoop-hive-2.1.0.Beta4.jar]
Norows affected (0.122seconds)
|
Then I am able to create an external table:
|
CREATE EXTERNAL table estest (field STRING)
STORED BY 'org.elasticsearch.hadoop.hive.EsStorageHandler'
TBLPROPERTIES('es.resource'='hadoop/hadoop','es.index.auto.create'='false');
As you can see I've followed the recommendation to put the jar file in HDFS, and it seems like the jar is picked up in
the classpath since without the 'add jar' I get another error stating that the EsStorageHandler can't be found. Any
clues as to why this is happening?
Thanks. We got it working by adding the jar to the hive-config, rather than
by "add jar" ..
-ra
Den onsdag 29 april 2015 kl. 00:11:47 UTC+2 skrev Costin Leau:
Hi,
It seems you are running into a classpath problem. The class mentioned in
the exception (org/elasticsearch/hadoop/serialization/dto/Node) is part of
the elasticsearch-hadoop-hive-XXX. jar - you can verify this yourself.
The fact that it is not found at runtime suggests that the a different or
incomplete jar is used instead. This can occur for example if a different
jar is available in the Hive/Hadoop classpath which is picked up
automatically and overrides the one you use in your script.
So first try and double check the existing classpath - in the vast
majority of Hive problems, this was the issue (and old version was picked
up instead). You can also verify this by trying to register the table - you
should get an exception right away. Once that's done, try different ways of
adding the jar to your script classpath - it might be that beeline has a
different mechanism than vanilla Hive.
Hope this helps,
On 4/29/15 12:58 AM, Rasmus Aveskogh wrote:
Hi!
I've followed the various guides to get going with the
elasticsearch-hadoop-integration in Hive, but I run into some issue:
add jar hdfs:
//host:9000//lib/elasticsearch-hadoop-hive-2.1.0.Beta4.jar;
INFO : converting to local hdfs:
//host:9000//lib/elasticsearch-hadoop-hive-2.1.0.Beta4.jar
INFO : Added [/tmp/15207d6b-e4b5-446b-bbe2-cff282056983_resources/
elasticsearch-hadoop-hive-2.1.0.Beta4.jar] to class path
INFO : Added resources: [hdfs:
//host:9000//lib/elasticsearch-hadoop-hive-2.1.0.Beta4.jar]
select * from estest;
Error: java.lang.NoClassDefFoundError: org/elasticsearch/hadoop/
serialization/dto/Node (state=,code=0)
As you can see I've followed the recommendation to put the jar file in
HDFS, and it seems like the jar is picked up in the classpath since without
the 'add jar' I get another error stating that the EsStorageHandler can't
be found. Any clues as to why this is happening?
Glad to hear it got solved. Btw, what version of hive/beeline/hadoop are
you using?
On Apr 29, 2015 3:18 PM, "Rasmus Aveskogh" aveskogh@gmail.com wrote:
Thanks. We got it working by adding the jar to the hive-config, rather
than by "add jar" ..
-ra
Den onsdag 29 april 2015 kl. 00:11:47 UTC+2 skrev Costin Leau:
Hi,
It seems you are running into a classpath problem. The class mentioned in
the exception (org/elasticsearch/hadoop/serialization/dto/Node) is part of
the elasticsearch-hadoop-hive-XXX. jar - you can verify this yourself.
The fact that it is not found at runtime suggests that the a different or
incomplete jar is used instead. This can occur for example if a different
jar is available in the Hive/Hadoop classpath which is picked up
automatically and overrides the one you use in your script.
So first try and double check the existing classpath - in the vast
majority of Hive problems, this was the issue (and old version was picked
up instead). You can also verify this by trying to register the table - you
should get an exception right away. Once that's done, try different ways of
adding the jar to your script classpath - it might be that beeline has a
different mechanism than vanilla Hive.
Hope this helps,
On 4/29/15 12:58 AM, Rasmus Aveskogh wrote:
Hi!
I've followed the various guides to get going with the
elasticsearch-hadoop-integration in Hive, but I run into some issue:
add jar hdfs:
//host:9000//lib/elasticsearch-hadoop-hive-2.1.0.Beta4.jar;
INFO : converting to local hdfs:
//host:9000//lib/elasticsearch-hadoop-hive-2.1.0.Beta4.jar
INFO : Added [/tmp/15207d6b-e4b5-446b-bbe2-cff282056983_resources/
elasticsearch-hadoop-hive-2.1.0.Beta4.jar] to class path
INFO : Added resources: [hdfs:
//host:9000//lib/elasticsearch-hadoop-hive-2.1.0.Beta4.jar]
select * from estest;
Error: java.lang.NoClassDefFoundError: org/elasticsearch/hadoop/
serialization/dto/Node (state=,code=0)
As you can see I've followed the recommendation to put the jar file in
HDFS, and it seems like the jar is picked up in the classpath since without
the 'add jar' I get another error stating that the EsStorageHandler can't
be found. Any clues as to why this is happening?
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.