Exception while using elastic-hadoop library for Apache Spark from spark-shell

Elastic Stack Elasticsearch
1

Hi all,

I'm new to the world of apache spark/hadoop and I'm trying the simple examples i found on the elasticsearch/spark web page to simply read/write data to elasticsearch from the spark-shell.

I've tried to write data, and it worked just fine! When I read it works but then when i try to show the content of the RDD doing RDD.collect() it throws and exception:

scala> RDD.collect()
17/06/21 14:26:50 ERROR Executor: Exception in task 0.0 in stage 3.0 (TID 13)
java.lang.NoClassDefFoundError: scala/collection/GenTraversableOnce$class
	at org.elasticsearch.spark.rdd.AbstractEsRDDIterator.<init>(AbstractEsRDDIterator.scala:28)
	at org.elasticsearch.spark.rdd.ScalaEsRDDIterator.<init>(ScalaEsRDD.scala:43)
	at org.elasticsearch.spark.rdd.ScalaEsRDD.compute(ScalaEsRDD.scala:39)
	at org.elasticsearch.spark.rdd.ScalaEsRDD.compute(ScalaEsRDD.scala:33)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
	at org.apache.spark.scheduler.Task.run(Task.scala:99)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:322)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:748)

I'm using Scala version 2.11.8, spark version 2.1.1 and elasticsearch-hadoop-5.4.1.jar.
thanks

2

This often occurs when using the incorrect Scala versions of Spark or elasticsearch-hadoop. Try using the 2.11 compatibility jar for Spark.

3

Thnks @james.baiera , it's frustating to find the correct jar that is not deprecated even if it was released on May...

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.