ERROR: Multiple ES-Hadoop versions detected

Elastic Stack Elasticsearch
1

Hi,
We are using Hadoop with elasticsearch on a cluster of 5 nodes and we are using : CDH 5.8.0, eastic search 2.3.5 and elasticsearch-hadoop-2.3.4.

We created a table on elastic search using hive and it worked and we are trying to load the data from hive :
INSERT overwrite TABLE cdr_es SELECT ....
FROM cdr_link

But we have the following error :

Diagnostic Messages for this Task:
Error: java.lang.RuntimeException: java.lang.Error: Multiple ES-Hadoop versions detected in the classpath; please use only one
jar:file:/data/yarn/nm/usercache/cloudera-scm/filecache/938/elasticsearch-hadoop-cascading-2.3.4.jar
jar:file:/data/yarn/nm/usercache/cloudera-scm/filecache/955/elasticsearch-hadoop-hive-2.3.4.jar
jar:file:/data/yarn/nm/usercache/cloudera-scm/filecache/931/elasticsearch-storm-2.3.4.jar
jar:file:/data/yarn/nm/usercache/cloudera-scm/filecache/940/elasticsearch-hadoop-mr-2.3.4.jar
jar:file:/data/yarn/nm/usercache/cloudera-scm/filecache/926/elasticsearch-spark_2.11-2.3.4.jar
jar:file:/data/yarn/nm/usercache/cloudera-scm/filecache/937/elasticsearch-spark-1.2_2.10-2.3.4.jar
jar:file:/data/yarn/nm/usercache/cloudera-scm/filecache/929/elasticsearch-spark-1.2_2.11-2.3.4.jar
jar:file:/data/yarn/nm/usercache/cloudera-scm/filecache/950/elasticsearch-hadoop-pig-2.3.4.jar
jar:file:/data/yarn/nm/usercache/cloudera-scm/filecache/959/elasticsearch-spark_2.10-2.3.4.jar
jar:file:/data/yarn/nm/usercache/cloudera-scm/filecache/943/elasticsearch-hadoop-2.3.4.jar

at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:179)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)

Caused by: java.lang.Error: Multiple ES-Hadoop versions detected in the classpath; please use only one

If we delete one of the jars (elasticsearch-hadoop-hive-2.3.4 or elasticsearch-hadoop-cascading-2.3.4.jar or elasticsearch-hadoop-2.3.4.jar) than we will have an error with a missing package.

How can we solve the problem ?
Thanks

2

You seem to be using all of the jars when you should be using only one. We provide the full package of features in one jar (elasticsearch-hadoop-2.3.4.jar), but we also provide dedicated jars for each technology should users desire to use them in a standalone fashion. Keep the elasticsearch-hadoop-2.3.4.jar and remove the others.

3

Hi James,
Thanks for the answer.
If we leave only the jar that we need (elasticsearch-hadoop-2.3.4.jar) then we will have another errors "File file:/usr/lib/elasticsearch-hadoop-2.3.4/dist/elasticsearch-hadoop-cascading-2.3.4.jar does not exist"

2016-08-30 08:20:54,773 ERROR org.apache.hadoop.hive.ql.exec.Task: [HiveServer2-Background-Pool: Thread-1040]: Job Submission failed with exception 'java.io.FileNotFoundException(File file:/usr/lib/elasticsearch-hadoop-2.3.4/dist/elasticsearch-hadoop-cascading-2.3.4.jar does not exist)'
java.io.FileNotFoundException: File file:/usr/lib/elasticsearch-hadoop-2.3.4/dist/elasticsearch-hadoop-cascading-2.3.4.jar does not exist

If we have all the libraries then the map works perfectly but we have the "Multiple ES-Hadoop versions detected" error when starting to reduce.

How can we solve that ?

Thanks

4

We used elasticsearch-hadoop-2.3.3.jar and it worked without problems.

I guess that there is a problem with the 2.3.4

Thank you