# ES-Hadoop on HDInsight cluster **URL:** \ **Category:** Elasticsearch\ **Tags:** es-hadoop\ **Created:** [July 12, 2017, 9:19pm UTC](https://discuss.elastic.co/t/es-hadoop-on-hdinsight-cluster/92879 "2017-07-12T21:19:56Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![arc](https://avatars.discourse-cdn.com/v4/letter/a/67e7ee/32.png) [@arc](https://discuss.elastic.co/u/arc)\ **Post date:** [July 12, 2017, 9:19pm UTC](https://discuss.elastic.co/t/es-hadoop-on-hdinsight-cluster/92879/1 "2017-07-12T21:19:56Z")
I am running a hive job that reads data from azure blob (represented as a hive external table) and writes to elasticsearch via **elasticsearch-hadoop-5.4.3.jar**. Namely, the first line of my hive script is ADD JAR ./elasticsearch-hadoop-5.4.3.jar; The job was unsuccessful, returning the attached messages. Is there anything extra I need to install or add on HDInsight cluster? Status: Failed Vertex failed, vertexName=Reducer 2, vertexId=vertex\_1499891279427\_0003\_1\_01, diagnostics=[Task failed, taskId=task\_1499891279427\_0003\_1\_01\_000000, diagnostics=[TaskAttempt 0 failed, info=[Error: Failure while running task:java.lang.RuntimeException: java.lang.RuntimeException: java.lang.NoClassDefFoundError: org/apache/commons/httpclient/protocol/ProtocolSocketFactory at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:173) at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:139) at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:347) at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:194) at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:185) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1865) at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:185) at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:181) at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.RuntimeException: java.lang.NoClassDefFoundError: org/apache/commons/httpclient/protocol/ProtocolSocketFactory at org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecordVector(ReduceRecordSource.java:390) at org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecord(ReduceRecordSource.java:232) at org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.run(ReduceRecordProcessor.java:266) at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:150) ... 14 more ... (omitted 30000 lines as they appears to presenting similar information, namely, having **NoClassDefFoundError** ) Vertex did not succeed due to OWN\_TASK\_FAILURE, failedTasks:1 killedTasks:0, Vertex vertex\_1499891279427\_0003\_1\_01 [Reducer 2] killed/failed due to:OWN\_TASK\_FAILURE]DAG did not succeed due to VERTEX\_FAILURE. failedVertices:1 killedVertices:0 ---
**Author:** ![james.baiera](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/james.baiera/32/10209_2.png) [@james.baiera](https://discuss.elastic.co/u/james.baiera)\ **Post date:** [July 17, 2017, 2:09am UTC](https://discuss.elastic.co/t/es-hadoop-on-hdinsight-cluster/92879/2 "2017-07-17T02:09:39Z")
@arc do you have apache commons http client available on the job's class path? Could you include the versions of the relevant frameworks you are using? ---
**Author:** ![arc](https://avatars.discourse-cdn.com/v4/letter/a/67e7ee/32.png) [@arc](https://discuss.elastic.co/u/arc)\ **Post date:** [July 17, 2017, 5:27am UTC](https://discuss.elastic.co/t/es-hadoop-on-hdinsight-cluster/92879/3 "2017-07-17T05:27:39Z")
Yes I do. Turns out that HDInsight uses Hortonwork Hadoop and the http client can be accessed with: ADD JAR /usr/hdp/current/hive-client/lib/commons-httpclient-3.0.1.jar; ---
**Author:** ![james.baiera](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/james.baiera/32/10209_2.png) [@james.baiera](https://discuss.elastic.co/u/james.baiera)\ **Post date:** [July 17, 2017, 3:31pm UTC](https://discuss.elastic.co/t/es-hadoop-on-hdinsight-cluster/92879/4 "2017-07-17T15:31:20Z")
Could you include the complete stack trace? ---
**Author:** ![arc](https://avatars.discourse-cdn.com/v4/letter/a/67e7ee/32.png) [@arc](https://discuss.elastic.co/u/arc)\ **Post date:** [July 21, 2017, 9:11pm UTC](https://discuss.elastic.co/t/es-hadoop-on-hdinsight-cluster/92879/5 "2017-07-21T21:11:24Z")
Hi James, I cannot post the whole stack because a post is limited to 7000 characters (the whole stack has 35000+ characters) . Here is an excerpt FYI: Status: Failed Vertex failed, vertexName=Map 1, vertexId=vertex\_1500669401375\_0003\_1\_00, diagnostics=[Task failed, taskId=task\_1500669401375\_0003\_1\_00\_000027, diagnostics=[TaskAttempt 0 failed, info=[Error: Failure while running task:java.lang.RuntimeException: java.lang.RuntimeException: java.lang.NoClassDefFoundError: org/apache/commons/httpclient/Credentials at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:173) at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:139) at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:347) at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:194) at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:185) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1865) at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:185) at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:181) at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.RuntimeException: java.lang.NoClassDefFoundError: org/apache/commons/httpclient/Credentials at org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:91) at org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:68) at org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.run(MapRecordProcessor.java:325) at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:150) ... 14 more Caused by: java.lang.NoClassDefFoundError: org/apache/commons/httpclient/Credentials at org.elasticsearch.hadoop.rest.commonshttp.CommonsHttpTransportFactory.create(CommonsHttpTransportFactory.java:39) at org.elasticsearch.hadoop.rest.NetworkClient.selectNextNode(NetworkClient.java:99) at org.elasticsearch.hadoop.rest.NetworkClient.(NetworkClient.java:82) at org.elasticsearch.hadoop.rest.NetworkClient.(NetworkClient.java:59) at org.elasticsearch.hadoop.rest.RestClient.(RestClient.java:92) at org.elasticsearch.hadoop.rest.InitializationUtils.discoverNodesIfNeeded(InitializationUtils.java:87) at org.elasticsearch.hadoop.rest.RestService.createWriter(RestService.java:574) at org.elasticsearch.hadoop.mr.EsOutputFormat$EsRecordWriter.init(EsOutputFormat.java:173) at org.elasticsearch.hadoop.hive.EsHiveOutputFormat$EsHiveRecordWriter.write(EsHiveOutputFormat.java:58) at org.apache.hadoop.hive.ql.exec.FileSinkOperator.process(FileSinkOperator.java:764) at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:841) at org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:88) at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:841) at org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:133) at org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.forward(MapOperator.java:170) at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:555) at org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:83) ... 17 more Caused by: java.lang.ClassNotFoundException: org.apache.commons.httpclient.Credentials at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ... 34 more ], TaskAttempt 1 failed, info=[Error: Failure while running task:java.lang.RuntimeException: TaskAttempt 3 failed, info=[Error: Failure while running task:java.lang.RuntimeException: java.lang.RuntimeException: java.lang.NoClassDefFoundError: org/apache/commons/httpclient/protocol/ProtocolSocketFactory at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:173) at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:139) at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:347) at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:194) at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:185) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1865) at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:185) at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:181) at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.RuntimeException: java.lang.NoClassDefFoundError: org/apache/commons/httpclient/protocol/ProtocolSocketFactory at org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:91) at org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:68) at org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.run(MapRecordProcessor.java:325) at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:150) ... 14 more ]], Vertex did not succeed due to OWN\_TASK\_FAILURE, failedTasks:1 killedTasks:39, Vertex vertex\_1500669401375\_0003\_1\_00 [Map 1] killed/failed due to:OWN\_TASK\_FAILURE]DAG did not succeed due to VERTEX\_FAILURE. failedVertices:1 killedVertices:0 ---
**Author:** ![arc](https://avatars.discourse-cdn.com/v4/letter/a/67e7ee/32.png) [@arc](https://discuss.elastic.co/u/arc)\ **Post date:** [July 21, 2017, 9:12pm UTC](https://discuss.elastic.co/t/es-hadoop-on-hdinsight-cluster/92879/6 "2017-07-21T21:12:58Z")
Probably I was not clear in the previous post that the problem is resolved by adding: ADD JAR /usr/hdp/current/hive-client/lib/commons-httpclient-3.0.1.jar; If you have additional comments/tips please let me know! ---
**Author:** ![james.baiera](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/james.baiera/32/10209_2.png) [@james.baiera](https://discuss.elastic.co/u/james.baiera)\ **Post date:** [July 21, 2017, 9:30pm UTC](https://discuss.elastic.co/t/es-hadoop-on-hdinsight-cluster/92879/7 "2017-07-21T21:30:22Z")
That is fairly strange. We depend on commons-httpclient-3.0.1 for the underlying rest client. We chose to use that library since it was pre-packaged and available in most Hadoop distributions. It's possible that HDInsights has removed their usage of that jar. The library is removed in Hadoop 2.8, so it would not surprise me if the underlying Hadoop version you are on is based on that, or that it was removed in preparation for 2.8. If re-adding the missing jar has fixed the problem, then I'll mark that as the solution to the post. ---
**Author:** ![arc](https://avatars.discourse-cdn.com/v4/letter/a/67e7ee/32.png) [@arc](https://discuss.elastic.co/u/arc)\ **Post date:** [July 21, 2017, 9:46pm UTC](https://discuss.elastic.co/t/es-hadoop-on-hdinsight-cluster/92879/8 "2017-07-21T21:46:30Z")
typing "hadoop version" in the command line shows: Hadoop 2.7.3.2.5.5.3-2 Subversion [git@github.com](mailto:git@github.com):hortonworks/hadoop.git -r dff5d68bdb3763da7dc4c3f02666ce2f3ec28a91 Compiled by jenkins on 2017-05-23T02:10Z Compiled with protoc 2.5.0 From source with checksum a2161cab5fb2bc8e577c41447ea1f8 This command was run using /usr/hdp/2.5.5.3-2/hadoop/hadoop-common-2.7.3.2.5.5.3-2.jar Yes, I confirm that adding the jar file solves the problem. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [August 18, 2017, 9:49pm UTC](https://discuss.elastic.co/t/es-hadoop-on-hdinsight-cluster/92879/9 "2017-08-18T21:49:51Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.