Problem integrating elasticsearch hadoop with Hortonworks sandbox

Elastic Stack Elasticsearch
1

Hi, I'm pretty new to Hadoop, and I'm following this tutorial to try to integrate elasticsearch to Hadoop. Everything went well util I tried to execute below statement:

INSERT OVERWRITE TABLE eslogs SELECT s.time, s.ext, s.ip, s.req, s.res, s.agent FROM logs s;

Exception was thrown.. java.lang.NoClassDefFoundError: org/apache/commons/httpclient/URIException..
I followed this post Hive (HDP 2.3) and ES-Hadoop Integration Issue to manually add the httpclient jar file in, but the error persisted. How can I fix this issue?

2

Hello!

Could you post which versions of the systems and artifacts you are using as well as the classpath from your executing hive tasks? Could you also post the commands you used to add the httpclient jar? Thanks!

3

Hi James,
I'm using Hortonworks Sandbox 2.4 for VMWare, and elasticsearch-hadoop 2.3.2
The elasticsearch version i'm using 2.3.3.

I copied all ES related JARs to the hive lib directory, used the following commands to add in the JARs
ADD JAR /usr/hdp/2.4.0.0-169/hive/lib/elasticsearch-hadoop-2.3.2.jar
ADD JAR /usr/hdp/2.4.0.0-169/hive/lib/httpclient-4.4.jar

If I do the LIST JARS command, I only see the above two JARs listed.

Thank you!

4

Hi, realised I have added in the wrong http lib, the error went away once I add in the common http client library, thank you.

5

Cheers!