Multiple ES-Hadoop versions detected in the classpath in Hue

Elastic Stack Elasticsearch
1

Hello,

We are using Cloudera CDH5.12 and elasticsearch-hadoop-6.2.1 . We added the ES-hadoop jars path to Cloudera Manager->Clusters->Hive->Advanced and set the value of: Hive Auxiliary JARs Directory to the jar directory. Now if we keep all the 21 jars for different tools like mr, hive, pig, spark etc. in the directory we get the below error.

Diagnostic Messages for this Task:
Error: java.lang.RuntimeException: java.lang.Error: Multiple ES-Hadoop versions detected in the classpath; please use only one
jar:file:/yarn/nm/usercache/hive/filecache/1747/elasticsearch-spark-20_2.11-6.2.1.jar
jar:file:/yarn/nm/usercache/hive/filecache/1751/elasticsearch-hadoop-mr-6.2.1.jar
jar:file:/yarn/nm/usercache/hive/filecache/1755/elasticsearch-hadoop-cascading-6.2.1.jar
jar:file:/yarn/nm/usercache/hive/filecache/1738/elasticsearch-hadoop-6.2.1.jar
jar:file:/yarn/nm/usercache/hive/filecache/1743/elasticsearch-hadoop-pig-6.2.1.jar
jar:file:/yarn/nm/usercache/hive/filecache/1739/elasticsearch-hadoop-hive-6.2.1.jar
jar:file:/yarn/nm/usercache/hive/filecache/1760/elasticsearch-storm-6.2.1.jar

    at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:179)
    at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
    at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:459)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
    at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:415)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1917)
    at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)

But if we remove all the other jars and only keep one jar as was suggested in other posts then we get the opposite error. This seems like a catch-22 situation where we can neither remove nor keep all the other jars. Why cant the code just ignore the jars not needed if its the same version?

2018-02-21 17:11:55,641 INFO org.apache.hadoop.hive.ql.exec.TableScanOperator: [HiveServer2-Handler-Pool: Thread-727]: Initialization Done 393 TS
2018-02-21 17:12:18,366 ERROR org.apache.hadoop.hive.common.FileUtils: [HiveServer2-Handler-Pool: Thread-728]: The jar file path file:///opt/elastichadoop/elasticsearch-hadoop-6.2.1/dist/elasticsearc
h-hadoop-cascading-6.2.1.jar doesn't exist
2018-02-21 17:12:18,366 ERROR org.apache.hadoop.hive.common.FileUtils: [HiveServer2-Handler-Pool: Thread-728]: The jar file path file:///opt/elastichadoop/elasticsearch-hadoop-6.2.1/dist/elasticsearc
h-hadoop-cascading-6.2.1-javadoc.jar doesn't exist
2018-02-21 17:12:18,366 ERROR org.apache.hadoop.hive.common.FileUtils: [HiveServer2-Handler-Pool: Thread-728]: The jar file path file:///opt/elastichadoop/elasticsearch-hadoop-6.2.1/dist/elasticsearc
h-hadoop-cascading-6.2.1-sources.jar doesn't exist
2018-02-21 17:12:18,366 ERROR org.apache.hadoop.hive.common.FileUtils: [HiveServer2-Handler-Pool: Thread-728]: The jar file path file:///opt/elastichadoop/elasticsearch-hadoop-6.2.1/dist/elasticsearc
h-hadoop-mr-6.2.1.jar doesn't exist
2018-02-21 17:12:18,366 ERROR org.apache.hadoop.hive.common.FileUtils: [HiveServer2-Handler-Pool: Thread-728]: The jar file path file:///opt/elastichadoop/elasticsearch-hadoop-6.2.1/dist/elasticsearc
h-hadoop-mr-6.2.1-javadoc.jar doesn't exist
2018-02-21 17:12:18,366 ERROR org.apache.hadoop.hive.common.FileUtils: [HiveServer2-Handler-Pool: Thread-728]: The jar file path file:///opt/elastichadoop/elasticsearch-hadoop-6.2.1/dist/elasticsearc
h-hadoop-mr-6.2.1-sources.jar doesn't exist
2018-02-21 17:12:18,366 ERROR org.apache.hadoop.hive.common.FileUtils: [HiveServer2-Handler-Pool: Thread-728]: The jar file path file:///opt/elastichadoop/elasticsearch-hadoop-6.2.1/dist/elasticsearc
h-hadoop-pig-6.2.1.jar doesn't exist
2018-02-21 17:12:18,366 ERROR org.apache.hadoop.hive.common.FileUtils: [HiveServer2-Handler-Pool: Thread-728]: The jar file path file:///opt/elastichadoop/elasticsearch-hadoop-6.2.1/dist/elasticsearc
h-hadoop-pig-6.2.1-javadoc.jar doesn't exist
2018-02-21 17:12:18,367 ERROR org.apache.hadoop.hive.common.FileUtils: [HiveServer2-Handler-Pool: Thread-728]: The jar file path file:///opt/elastichadoop/elasticsearch-hadoop-6.2.1/dist/elasticsearch-hadoop-pig-6.2.1-sources.jar doesn't exist
2018-02-21 17:12:18,367 ERROR org.apache.hadoop.hive.common.FileUtils: [HiveServer2-Handler-Pool: Thread-728]: The jar file path file:///opt/elastichadoop/elasticsearch-hadoop-6.2.1/dist/elasticsearch-spark-20_2.11-6.2.1.jar doesn't exist
2018-02-21 17:12:18,367 ERROR org.apache.hadoop.hive.common.FileUtils: [HiveServer2-Handler-Pool: Thread-728]: The jar file path file:///opt/elastichadoop/elasticsearch-hadoop-6.2.1/dist/elasticsearch-spark-20_2.11-6.2.1-javadoc.jar doesn't exist
2018-02-21 17:12:18,367 ERROR org.apache.hadoop.hive.common.FileUtils: [HiveServer2-Handler-Pool: Thread-728]: The jar file path file:///opt/elastichadoop/elasticsearch-hadoop-6.2.1/dist/elasticsearch-spark-20_2.11-6.2.1-sources.jar doesn't exist
2018-02-21 17:12:18,367 ERROR org.apache.hadoop.hive.common.FileUtils: [HiveServer2-Handler-Pool: Thread-728]: The jar file path file:///opt/elastichadoop/elasticsearch-hadoop-6.2.1/dist/elasticsearch-storm-6.2.1.jar doesn't exist
2018-02-21 17:12:18,367 ERROR org.apache.hadoop.hive.common.FileUtils: [HiveServer2-Handler-Pool: Thread-728]: The jar file path file:///opt/elastichadoop/elasticsearch-hadoop-6.2.1/dist/elasticsearch-storm-6.2.1-javadoc.jar doesn't exist
2018-02-21 17:12:18,367 ERROR org.apache.hadoop.hive.common.FileUtils: [HiveServer2-Handler-Pool: Thread-728]: The jar file path file:///opt/elastichadoop/elasticsearch-hadoop-6.2.1/dist/elasticsearch-storm-6.2.1-sources.jar doesn't exist
2018-02-21 17:12:18,368 WARN org.apache.hadoop.hive.conf.HiveConf: [HiveServer2-Handler-Pool: Thread-728]: HiveConf of name hive.server2.idle.session.timeout_check_operation does not exist

2

This is the only jar that you should need if you want to use all of the integrations. Can you give some more information about the errors that you are seeing after you remove the jars? What are you trying to run, can you include the code, etc...

1 Like
3

Hello James, appreciate the response.
So far all my efforts to try and get it working has failed. The only time it worked is by adding the jar directly in Hive cli. For example sampledata1 is a hive external table which is defined as ES index:
hive> ADD JAR file:///opt/elastichadoop/elasticsearch-hadoop-6.2.1/dist/elasticsearch-hadoop-6.2.1.jar;
Added [file:///opt/elastichadoop/elasticsearch-hadoop-6.2.1/dist/elasticsearch-hadoop-6.2.1.jar] to class path
Added resources: [file:///opt/elastichadoop/elasticsearch-hadoop-6.2.1/dist/elasticsearch-hadoop-6.2.1.jar]
hive> select * from sampledata1;
OK
1 100 msg line1 data1
2 200 msg line2 data2
Time taken: 2.789 seconds, Fetched: 2 row(s)

Next as stated in the initial post, I used Cloudera Manager 5.12 to add the Cloudera Manager->Clusters->Hive->Advanced and set the value of: Hive Auxiliary JARs Directory to the jar directory .
In this scenario if I use Hadoop Hue tool or other JDBC query tools like MicroStrategy to query the external table I will get the errors given in the initial post complaining about either all the different jars are needed or else if I add all the jars to the directory it will say multiple ES-Hadoop versions detected in the classpath use only one. If you have access to Cloudera Manager and Hue or download the free MicroStrategy desktop you can verify by querying after adding to the Hive Auxiliary JARs Directory.

4

I do not have much experience with using Hue unfortunately. This sounds like an issue that perhaps someone from Cloudera might have better insight into fixing. In the project we do all of our testing against a locally running Hive server instance, and the jars are all configured via the ADD JAR mechanism. If you are able to make any progress, we'd love if you posted more about it here for other users!

5

Thanks for the info. Yes will continue checking the issue in Cloudera or Hue area and see what could be causing this.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.