Multiple ES-Hadoop versions detected in the classpath

Elastic Stack Elasticsearch
1

Getting Multiple ES-Hadoop versions issues. Using MapR distribution of Hadoop. Running Spark job in yarn/cluster mode.

java.lang.Error: Multiple ES-Hadoop versions detected in the classpath; please use only one
jar:file:/tmp/hadoop-accion_mapr/nm-local-dir/usercache/bprabhakar/appcache/application_1486664324019_1027/container_e36_1486664324019_1027_01_000002/elasticsearch-spark-20_2.11-5.0.0.jar
jar:file:/mapr/airavata/local/h10n5.accion.com/tmp/hadoop/nm-local-dir/usercache/bprabhakar/appcache/application_1486664324019_1027/container_e36_1486664324019_1027_01_000002/elasticsearch-spark-20_2.11-5.0.0.jar

            at org.elasticsearch.hadoop.util.Version.<clinit>(Version.java:73)
            at org.elasticsearch.hadoop.rest.RestService.createWriter(RestService.java:542)
            at org.elasticsearch.spark.rdd.EsRDDWriter.write(EsRDDWriter.scala:58)
            at org.elasticsearch.spark.rdd.EsSpark$$anonfun$doSaveToEs$1.apply(EsSpark.scala:102)
            at org.elasticsearch.spark.rdd.EsSpark$$anonfun$doSaveToEs$1.apply(EsSpark.scala:102)
            at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:70)
            at org.apache.spark.scheduler.Task.run(Task.scala:86)
            at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274)
            at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
            at java.lang.Thread.run(Thread.java:745)
2

@G_Babu_Prabhakar It seems that you have multiple paths linking up to what looks like the same place on the filesystem:

/mapr/airavata/local/h10n5.accion.com/tmp/hadoop/
/tmp/hadoop-accion_mapr/

ES-Hadoop is probably sensing that there are multiple paths to the same artifact file. That or something is amiss with symlinks or mounting on your nodes...

3

Thanks James.

Any workaround to run the code till the actual issue is resolved?

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.