# Elastic Stack 5.4.0 is breaking for Spark 2.1.1

**URL:** <https://discuss.elastic.co/t/elastic-stack-5-4-0-is-breaking-for-spark-2-1-1/87627>\
**Category:** Elasticsearch\
**Tags:** es-hadoop\
**Created:** [May 30, 2017, 7:45pm UTC](https://discuss.elastic.co/t/elastic-stack-5-4-0-is-breaking-for-spark-2-1-1/87627 "2017-05-30T19:45:53Z")\
**Posts on this page:** 13\
**Page:** 1

<div class="post-metadata">

**Author:** ![Anupam\_Jaiswal1](https://avatars.discourse-cdn.com/v4/letter/a/4da419/32.png) [@Anupam\_Jaiswal1](https://discuss.elastic.co/u/Anupam_Jaiswal1)\
**Post date:** [May 30, 2017, 7:45pm UTC](https://discuss.elastic.co/t/elastic-stack-5-4-0-is-breaking-for-spark-2-1-1/87627/1 "2017-05-30T19:45:53Z")

</div>

Does the current library support spark 2.1.1 .  
The code is breaking for me . Printschema works fine , but show and collect breaks.

---

<div class="post-metadata">

**Author:** ![james.baiera](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/james.baiera/32/10209_2.png) [@james.baiera](https://discuss.elastic.co/u/james.baiera)\
**Post date:** [May 30, 2017, 10:35pm UTC](https://discuss.elastic.co/t/elastic-stack-5-4-0-is-breaking-for-spark-2-1-1/87627/2 "2017-05-30T22:35:02Z")

</div>

Could you link the logs/stacktraces that you are seeing? Have you upgraded from a previous version or is this a fresh install?

---

<div class="post-metadata">

**Author:** ![Anupam\_Jaiswal1](https://avatars.discourse-cdn.com/v4/letter/a/4da419/32.png) [@Anupam\_Jaiswal1](https://discuss.elastic.co/u/Anupam_Jaiswal1)\
**Post date:** [May 31, 2017, 4:09pm UTC](https://discuss.elastic.co/t/elastic-stack-5-4-0-is-breaking-for-spark-2-1-1/87627/3 "2017-05-31T16:09:49Z")

</div>

James

I did fresh installation of Spark 2.1.1 and installed ELK 5.4 and EShadoop  
5.4. Previuous version I had issues with reading child array into dataframe  
. I can printschema but but select crashes.

Have upgrade to the lastest version of spark and elk i am unable to run  
basic select from dataframe. please refer below stack.  
Traceback (most recent call last):  
File "/Users/anupamjaiswal/Documents/aj/pyspk.py", line 28, in   
df.select("tags").show()  
File  
"/usr/local/Cellar/apache-spark/2.1.1/libexec/python/lib/pyspark.zip/pyspark/sql/dataframe.py",  
line 318, in show  
print(self.\_jdf.showString(n, 20))  
File  
"/usr/local/Cellar/apache-spark/2.1.1/libexec/python/lib/py4j-0.10.4-src.zip/py4j/java\_gateway.py",  
line 1133, in **call**  
answer, self.gateway\_client, self.target\_id, [self.name](http://self.name))  
File  
"/usr/local/Cellar/apache-spark/2.1.1/libexec/python/lib/pyspark.zip/pyspark/sql/utils.py",  
line 63, in deco  
return f(\*a, \*\*kw)  
File  
"/usr/local/Cellar/apache-spark/2.1.1/libexec/python/lib/py4j-0.10.4-src.zip/py4j/protocol.py",  
line 319, in get\_return\_value  
format(target\_id, ".", name), value)  
Py4JJavaError: An error occurred while calling o37.showString.  
: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0  
in stage 0.0 failed 1 times, most recent failure: Lost task 0.0 in stage  
0.0 (TID 0, localhost, executor driver): java.lang.NoClassDefFoundError:  
scala/collection/GenTraversableOnce$class  
at  
org.elasticsearch.spark.rdd.AbstractEsRDDIterator.(AbstractEsRDDIterator.scala:28)  
at  
org.elasticsearch.spark.sql.ScalaEsRowRDDIterator.(ScalaEsRowRDD.scala:49)  
at org.elasticsearch.spark.sql.ScalaEsRowRDD.compute(ScalaEsRowRDD.scala:45)  
at org.elasticsearch.spark.sql.ScalaEsRowRDD.compute(ScalaEsRowRDD.scala:38)  
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)  
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)

---

<div class="post-metadata">

**Author:** ![james.baiera](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/james.baiera/32/10209_2.png) [@james.baiera](https://discuss.elastic.co/u/james.baiera)\
**Post date:** [May 31, 2017, 4:12pm UTC](https://discuss.elastic.co/t/elastic-stack-5-4-0-is-breaking-for-spark-2-1-1/87627/4 "2017-05-31T16:12:52Z")

</div>

Which version of Scala are you running Spark on top of and which specific version of ES-Hadoop are you using? Generally seeing `java.lang.NoClassDefFoundError: scala/collection/GenTraversableOnce$class` means that you are mismatching scala versions 2.10 and 2.11.

---

<div class="post-metadata">

**Author:** ![Anupam\_Jaiswal1](https://avatars.discourse-cdn.com/v4/letter/a/4da419/32.png) [@Anupam\_Jaiswal1](https://discuss.elastic.co/u/Anupam_Jaiswal1)\
**Post date:** [May 31, 2017, 5:01pm UTC](https://discuss.elastic.co/t/elastic-stack-5-4-0-is-breaking-for-spark-2-1-1/87627/5 "2017-05-31T17:01:53Z")

</div>

We are using pyspark for coding and have Scala version 2.11.8

---

<div class="post-metadata">

**Author:** ![Anupam\_Jaiswal1](https://avatars.discourse-cdn.com/v4/letter/a/4da419/32.png) [@Anupam\_Jaiswal1](https://discuss.elastic.co/u/Anupam_Jaiswal1)\
**Post date:** [June 1, 2017, 6:23pm UTC](https://discuss.elastic.co/t/elastic-stack-5-4-0-is-breaking-for-spark-2-1-1/87627/6 "2017-06-01T18:23:58Z")

</div>

Do we have any update ?

---

<div class="post-metadata">

**Author:** ![james.baiera](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/james.baiera/32/10209_2.png) [@james.baiera](https://discuss.elastic.co/u/james.baiera)\
**Post date:** [June 6, 2017, 4:21pm UTC](https://discuss.elastic.co/t/elastic-stack-5-4-0-is-breaking-for-spark-2-1-1/87627/7 "2017-06-06T16:21:30Z")

</div>

Sorry for disappearing there. This error almost always means that you are using the incorrect version of ES-Hadoop for your version of Scala. Do you have the artifact name for elasticsearch-spark that you are using? It should end in `_2.11`

---

<div class="post-metadata">

**Author:** ![Anupam\_Jaiswal1](https://avatars.discourse-cdn.com/v4/letter/a/4da419/32.png) [@Anupam\_Jaiswal1](https://discuss.elastic.co/u/Anupam_Jaiswal1)\
**Post date:** [June 7, 2017, 8:18pm UTC](https://discuss.elastic.co/t/elastic-stack-5-4-0-is-breaking-for-spark-2-1-1/87627/8 "2017-06-07T20:18:08Z")

</div>

My Bad was using wrong version . Do you have any working example of  
exploding nested json stored in elastic , will appreciate your help.

---

<div class="post-metadata">

**Author:** ![byakuinss](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/byakuinss/32/23954_2.png) [@byakuinss](https://discuss.elastic.co/u/byakuinss)\
**Post date:** [June 13, 2017, 2:25am UTC](https://discuss.elastic.co/t/elastic-stack-5-4-0-is-breaking-for-spark-2-1-1/87627/9 "2017-06-13T02:25:21Z")

</div>

I have the same problem, my Scala version and elasticsearch-spark are both in 2.11.  
I can successfully insert data to elasticsearch, but I can't read data from elasticsearch.  
(with Elastic Stack 5.4.0 and Spark 2.1.1)

My code:

> val es\_df = sqc.read.format("org.elasticsearch.spark.sql").load("test/daily\_player\_game")

My Maven Dependencies:

```
		<!-- https://mvnrepository.com/artifact/org.apache.spark/spark-core_2.11 -->
		<dependency>
			<groupId>org.apache.spark</groupId>
			<artifactId>spark-core_2.11</artifactId>
			<version>2.1.1</version>
		</dependency>

		<dependency>
			<groupId>org.apache.spark</groupId>
			<artifactId>spark-streaming_2.11</artifactId>
			<version>2.1.1</version>
		</dependency>

		<!-- https://mvnrepository.com/artifact/org.elasticsearch/elasticsearch-hadoop -->
		<dependency>
			<groupId>org.elasticsearch</groupId>
			<artifactId>elasticsearch-spark-20_2.11</artifactId>
			<version>5.3.1</version>
		</dependency>

```

Error Message:

> ```
> 17/06/13 10:01:04 WARN TaskSetManager: Lost task 0.0 in stage 0.0 (TID 0, 192.168.1.183, executor 2): java.lang.NoClassDefFoundError: scala/collection/GenTraversableOnce$class
> at org.elasticsearch.spark.rdd.AbstractEsRDDIterator.<init>(AbstractEsRDDIterator.scala:28)
> at org.elasticsearch.spark.sql.ScalaEsRowRDDIterator.<init>(ScalaEsRowRDD.scala:49)
> at org.elasticsearch.spark.sql.ScalaEsRowRDD.compute(ScalaEsRowRDD.scala:45)
> at org.elasticsearch.spark.sql.ScalaEsRowRDD.compute(ScalaEsRowRDD.scala:38)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
> at org.apache.spark.scheduler.Task.run(Task.scala:99)
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:322)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.ClassNotFoundException: scala.collection.GenTraversableOnce$class
> at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> 
> ```

---

<div class="post-metadata">

**Author:** ![james.baiera](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/james.baiera/32/10209_2.png) [@james.baiera](https://discuss.elastic.co/u/james.baiera)\
**Post date:** [June 22, 2017, 7:23pm UTC](https://discuss.elastic.co/t/elastic-stack-5-4-0-is-breaking-for-spark-2-1-1/87627/10 "2017-06-22T19:23:56Z")

</div>

@byakuinss, are you sure that there are no conflicting jars on your cluster?

---

<div class="post-metadata">

**Author:** ![byakuinss](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/byakuinss/32/23954_2.png) [@byakuinss](https://discuss.elastic.co/u/byakuinss)\
**Post date:** [June 26, 2017, 8:21am UTC](https://discuss.elastic.co/t/elastic-stack-5-4-0-is-breaking-for-spark-2-1-1/87627/11 "2017-06-26T08:21:22Z")

</div>

Yes. There is only one node in my elk cluster and it's a new server which does not have any old version jars before.

---

<div class="post-metadata">

**Author:** ![TGx](https://avatars.discourse-cdn.com/v4/letter/t/ecae2f/32.png) [@TGx](https://discuss.elastic.co/u/TGx)\
**Post date:** [July 2, 2017, 5:20am UTC](https://discuss.elastic.co/t/elastic-stack-5-4-0-is-breaking-for-spark-2-1-1/87627/12 "2017-07-02T05:20:26Z")

</div>

Thank you infomation.  
Very good.

* * *

[Sbobet](https://goo.gl/X1Z3pc)

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 30, 2017, 5:20am UTC](https://discuss.elastic.co/t/elastic-stack-5-4-0-is-breaking-for-spark-2-1-1/87627/13 "2017-07-30T05:20:44Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.