# Crash when reading DataFrame

**URL:** <https://discuss.elastic.co/t/crash-when-reading-dataframe/58511>\
**Category:** Elasticsearch\
**Tags:** es-hadoop\
**Created:** [August 20, 2016, 5:06pm UTC](https://discuss.elastic.co/t/crash-when-reading-dataframe/58511 "2016-08-20T17:06:42Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![m-kiuchi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/m-kiuchi/32/15219_2.png) [@m-kiuchi](https://discuss.elastic.co/u/m-kiuchi)\
**Post date:** [August 20, 2016, 5:06pm UTC](https://discuss.elastic.co/t/crash-when-reading-dataframe/58511/1 "2016-08-20T17:06:42Z")

</div>

**Environment:**  
ES: 5.0.0\_alpha5  
Spark: 1.6.2 ( vanilla-precompiled spark-1.6.2-bin-hadoop2.6.tgz)  
Python: 2.6.6  
ES-Hadoop: 5.0.0-alpha5

**Symptom**  
Spark with python has succeed to load ES index, but cannot read dataframe and job has crashed.

Any advice ?

**Code**

```auto
#!/usr/bin/python.
# -*- coding: utf-8 -*-
import sys
from pyspark import SparkContext, SparkConf
from pyspark.sql import SQLContext
#
# Environment Variables
#
ESHOST="localhost"
ESINDEX="raw-part-2016.08.16/availability"
conf = SparkConf().setAppName("PythonES-Sample")
conf.set('es_nodes' , ESHOST)
conf.set('es_resource', ESINDEX)
conf.set('es_query', "?q=machine_name:m_03")
sc = SparkContext(conf=conf)
sqlContext = SQLContext(sc)

df = sqlContext.read.format("org.elasticsearch.spark.sql").load("raw-part-2016.08.16/availability")
df.printSchema() # It works.

ret = df.first() # crash at this function
print ret

```

**log**

```auto
[m-kiuchi@estest ~]$ ~/spark/bin/spark-submit --driver-class-path=/home/m-kiuchi/es-hadoop/dist/elasticsearch-hadoop-5.0.0-alpha5.jar pyspark-es-4.py
Using Spark's repl log4j profile: org/apache/spark/log4j-defaults-repl.properties
To adjust logging level use sc.setLogLevel("INFO")
16/08/21 01:51:29 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
root
 |-- parts_name: string (nullable = true)
 |-- machine_name: string (nullable = true)
 |-- total_rot_360deg: long (nullable = true)
 |-- unix_time: timestamp (nullable = true)

16/08/21 01:51:44 ERROR Executor: Exception in task 0.0 in stage 0.0 (TID 0)
java.lang.NoClassDefFoundError: scala/collection/GenTraversableOnce$class
        at org.elasticsearch.spark.rdd.AbstractEsRDDIterator.<init>(AbstractEsRDDIterator.scala:10)
        at org.elasticsearch.spark.sql.ScalaEsRowRDDIterator.<init>(ScalaEsRowRDD.scala:31)
        at org.elasticsearch.spark.sql.ScalaEsRowRDD.compute(ScalaEsRowRDD.scala:27)
        at org.elasticsearch.spark.sql.ScalaEsRowRDD.compute(ScalaEsRowRDD.scala:20)
        at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
        at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
        at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
        at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
        at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
        at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
        at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
        at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
        at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
        at org.apache.spark.scheduler.Task.run(Task.scala:89)
        at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:227)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.ClassNotFoundException: scala.collection.GenTraversableOnce$class
        at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        ... 18 more
(...snip...)
16/08/21 01:51:44 ERROR TaskSetManager: Task 0 in stage 0.0 failed 1 times; aborting job

```

---

<div class="post-metadata">

**Author:** ![james.baiera](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/james.baiera/32/10209_2.png) [@james.baiera](https://discuss.elastic.co/u/james.baiera)\
**Post date:** [August 22, 2016, 9:05pm UTC](https://discuss.elastic.co/t/crash-when-reading-dataframe/58511/2 "2016-08-22T21:05:17Z")

</div>

> [@m-kiuchi](#):
>
> java.lang.NoClassDefFoundError: scala/collection/GenTraversableOnce$class

This error normally occurs when working with mixed Scala versions (2.10 vs. 2.11). What version of Scala are you currently using? Are your artifact and framework versions compatible with that version?

---

<div class="post-metadata">

**Author:** ![m-kiuchi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/m-kiuchi/32/15219_2.png) [@m-kiuchi](https://discuss.elastic.co/u/m-kiuchi)\
**Post date:** [August 23, 2016, 10:55am UTC](https://discuss.elastic.co/t/crash-when-reading-dataframe/58511/3 "2016-08-23T10:55:51Z")

</div>

Thanks for reply. I'm using pre-compiled spark ver 1.6.2(for Hadoop 2.6). So I didn't care what version of scala I was using...

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 1:23pm UTC](https://discuss.elastic.co/t/crash-when-reading-dataframe/58511/4 "2017-07-06T13:23:36Z")

</div>