Multiple ES-Hadoop versions detected in the classpath

Getting Multiple ES-Hadoop versions issues. Using MapR distribution of Hadoop. Running Spark job in yarn/cluster mode.

java.lang.Error: Multiple ES-Hadoop versions detected in the classpath; please use only one
jar:file:/tmp/hadoop-accion_mapr/nm-local-dir/usercache/bprabhakar/appcache/application_1486664324019_1027/container_e36_1486664324019_1027_01_000002/elasticsearch-spark-20_2.11-5.0.0.jar
jar:file:/mapr/airavata/local/h10n5.accion.com/tmp/hadoop/nm-local-dir/usercache/bprabhakar/appcache/application_1486664324019_1027/container_e36_1486664324019_1027_01_000002/elasticsearch-spark-20_2.11-5.0.0.jar

            at org.elasticsearch.hadoop.util.Version.<clinit>(Version.java:73)
            at org.elasticsearch.hadoop.rest.RestService.createWriter(RestService.java:542)
            at org.elasticsearch.spark.rdd.EsRDDWriter.write(EsRDDWriter.scala:58)
            at org.elasticsearch.spark.rdd.EsSpark$$anonfun$doSaveToEs$1.apply(EsSpark.scala:102)
            at org.elasticsearch.spark.rdd.EsSpark$$anonfun$doSaveToEs$1.apply(EsSpark.scala:102)
            at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:70)
            at org.apache.spark.scheduler.Task.run(Task.scala:86)
            at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274)
            at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
            at java.lang.Thread.run(Thread.java:745)

@G_Babu_Prabhakar It seems that you have multiple paths linking up to what looks like the same place on the filesystem:

/mapr/airavata/local/h10n5.accion.com/tmp/hadoop/
/tmp/hadoop-accion_mapr/

ES-Hadoop is probably sensing that there are multiple paths to the same artifact file. That or something is amiss with symlinks or mounting on your nodes...

Thanks James.

Any workaround to run the code till the actual issue is resolved?

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.