Exception while using elastic-hadoop library for Apache Spark from spark-shell

Hi all,

I'm new to the world of apache spark/hadoop and I'm trying the simple examples i found on the elasticsearch/spark web page to simply read/write data to elasticsearch from the spark-shell.

I've tried to write data, and it worked just fine! When I read it works but then when i try to show the content of the RDD doing RDD.collect() it throws and exception:

scala> RDD.collect()
17/06/21 14:26:50 ERROR Executor: Exception in task 0.0 in stage 3.0 (TID 13)
java.lang.NoClassDefFoundError: scala/collection/GenTraversableOnce$class
	at org.elasticsearch.spark.rdd.AbstractEsRDDIterator.<init>(AbstractEsRDDIterator.scala:28)
	at org.elasticsearch.spark.rdd.ScalaEsRDDIterator.<init>(ScalaEsRDD.scala:43)
	at org.elasticsearch.spark.rdd.ScalaEsRDD.compute(ScalaEsRDD.scala:39)
	at org.elasticsearch.spark.rdd.ScalaEsRDD.compute(ScalaEsRDD.scala:33)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
	at org.apache.spark.scheduler.Task.run(Task.scala:99)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:322)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:748)

I'm using Scala version 2.11.8, spark version 2.1.1 and elasticsearch-hadoop-5.4.1.jar.

This often occurs when using the incorrect Scala versions of Spark or elasticsearch-hadoop. Try using the 2.11 compatibility jar for Spark.

Thnks @james.baiera , it's frustating to find the correct jar that is not deprecated even if it was released on May...

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.