# Spark: java.lang.ClassCastException org.elasticsearch.spark.sql.EsDataFrameWriter to field org.elasticsearch.spark.sql.EsSparkSQL

**URL:** <https://discuss.elastic.co/t/spark-java-lang-classcastexception-org-elasticsearch-spark-sql-esdataframewriter-to-field-org-elasticsearch-spark-sql-essparksql/90073>\
**Category:** Elasticsearch\
**Tags:** es-hadoop\
**Created:** [June 20, 2017, 10:13am UTC](https://discuss.elastic.co/t/spark-java-lang-classcastexception-org-elasticsearch-spark-sql-esdataframewriter-to-field-org-elasticsearch-spark-sql-essparksql/90073 "2017-06-20T10:13:39Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![ebuildy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ebuildy/32/6070_2.png) [@ebuildy](https://discuss.elastic.co/u/ebuildy)\
**Post date:** [June 20, 2017, 10:13am UTC](https://discuss.elastic.co/t/spark-java-lang-classcastexception-org-elasticsearch-spark-sql-esdataframewriter-to-field-org-elasticsearch-spark-sql-essparksql/90073/1 "2017-06-20T10:13:39Z")

</div>

Running es4Hadoop 2.4.4 with Spark 1.6.0, Java 1.8, when:

`JavaEsSparkSQL.saveToEs(DATA_FRAME, "consolidation-v2/items");`

(where DATA\_FRAME is coming from a Parquet file).

I got the exception:

```
Caused by: java.lang.ClassCastException: cannot assign instance of org.elasticsearch.spark.sql.EsDataFrameWriter to field org.elasticsearch.spark.sql.EsSparkSQL$$anonfun$saveToEs$1.eta$0$1$1 of type org.elasticsearch.spark.sql.EsSchemaRDDWriter in instance of org.elasticsearch.spark.sql.EsSparkSQL$$anonfun$saveToEs$1
	at java.io.ObjectStreamClass$FieldReflector.setObjFieldValues(ObjectStreamClass.java:2089)
	at java.io.ObjectStreamClass.setObjFieldValues(ObjectStreamClass.java:1261)
	at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2006)
	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1924)
	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)
	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
	at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2000)
	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1924)
	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)
	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
	at java.io.ObjectInputStream.readObject(ObjectInputStream.java:371)
	at org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:76)
	at org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:115)
	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
	at org.apache.spark.scheduler.Task.run(Task.scala:89)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745) 

```

But doing the same thing via spark-shell is working fine 😕

---

<div class="post-metadata">

**Author:** ![james.baiera](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/james.baiera/32/10209_2.png) [@james.baiera](https://discuss.elastic.co/u/james.baiera)\
**Post date:** [June 21, 2017, 2:33pm UTC](https://discuss.elastic.co/t/spark-java-lang-classcastexception-org-elasticsearch-spark-sql-esdataframewriter-to-field-org-elasticsearch-spark-sql-essparksql/90073/2 "2017-06-21T14:33:13Z")

</div>

That's fairly peculiar. Can you double check which versions of ES-Spark are getting picked up? I'm wondering if there's a mismatch of versions on your driver versus what is running on the cluster... `EsSchemaRDDWriter` is only in the spark 1.2 compatibility distribution. `EsDataFrameWriter` is used in spark 1.3 and up.

---

<div class="post-metadata">

**Author:** ![ebuildy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ebuildy/32/6070_2.png) [@ebuildy](https://discuss.elastic.co/u/ebuildy)\
**Post date:** [June 21, 2017, 3:02pm UTC](https://discuss.elastic.co/t/spark-java-lang-classcastexception-org-elasticsearch-spark-sql-esdataframewriter-to-field-org-elasticsearch-spark-sql-essparksql/90073/3 "2017-06-21T15:02:07Z")

</div>

Ya it sounds something like a mix between compile & run time.

Now I include the es4Hadoop at build time (see [https://medium.com/@thomasdecaux/setup-elasticsearch-hadoop-plugin-for-apache-spark-2b6f7a75f77d](https://medium.com/@thomasdecaux/setup-elasticsearch-hadoop-plugin-for-apache-spark-2b6f7a75f77d)) and no more trouble.

Thanks you,

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 19, 2017, 3:02pm UTC](https://discuss.elastic.co/t/spark-java-lang-classcastexception-org-elasticsearch-spark-sql-esdataframewriter-to-field-org-elasticsearch-spark-sql-essparksql/90073/4 "2017-07-19T15:02:12Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.