# Pyspark Dataframe Save Error

**URL:** <https://discuss.elastic.co/t/pyspark-dataframe-save-error/98277>\
**Category:** Elasticsearch\
**Tags:** es-hadoop\
**Created:** [August 24, 2017, 6:42pm UTC](https://discuss.elastic.co/t/pyspark-dataframe-save-error/98277 "2017-08-24T18:42:45Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![David\_Latham](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/david_latham/32/21473_2.png) [@David\_Latham](https://discuss.elastic.co/u/David_Latham)\
**Post date:** [August 24, 2017, 6:42pm UTC](https://discuss.elastic.co/t/pyspark-dataframe-save-error/98277/1 "2017-08-24T18:42:45Z")

</div>

i'm running spark 1.5 and used es-hadoop 2.4 to connect to ES 2.1.1 server. reads and writes worked fine. after upgrading ES to 5.3.1 and downloading es-hadoop 5.3.1 (spark is still at 1.5) the reads work fine, but when attempting to write i get an error:

DF = sqlContext.read.parquet('data/on\_time\_performance.parquet')

DF.write.format("org.elasticsearch.spark.sql")  
.option("es.resource","agile\_data\_science/on\_time\_performance")  
.mode("overwrite")  
.save()

Py4JJavaError: An error occurred while calling o39.save.  
: java.lang.AbstractMethodError: org.elasticsearch.spark.sql.DefaultSource.createRelation(Lorg/apache/spark/sql/SQLContext;Lorg/apache/spark/sql/SaveMode;Lscala/collection/immutable/Map;Lorg/apache/spark/sql/Dataset;)Lorg/apache/spark/sql/sources/BaseRelation;  
at org.apache.spark.sql.execution.datasources.DataSource.write(DataSource.scala:429)  
at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:211)  
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)  
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)  
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)  
at java.lang.reflect.Method.invoke(Method.java:497)  
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:237)  
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)  
at py4j.Gateway.invoke(Gateway.java:280)  
at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:128)  
at py4j.commands.CallCommand.execute(CallCommand.java:79)  
at py4j.GatewayConnection.run(GatewayConnection.java:211)  
at java.lang.Thread.run(Thread.java:745)

all the threads say to use the right driver, but which one? i've got about 10 downloaded from 2.1 - \> 5.3.1. none of the 2.X ones will work against 5.3.1 ES server, and all the 5.X drivers give the same error.

Thanks,

---

<div class="post-metadata">

**Author:** ![james.baiera](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/james.baiera/32/10209_2.png) [@james.baiera](https://discuss.elastic.co/u/james.baiera)\
**Post date:** [August 31, 2017, 3:23pm UTC](https://discuss.elastic.co/t/pyspark-dataframe-save-error/98277/2 "2017-08-31T15:23:23Z")

</div>

Since you're using spark 1.5 you need the 1.3-1.6 compatibility artifact for ES-Hadoop 5.3.1. Try `org.elasticsearch:elasticsearch-spark-13_2.10:5.3.1` or `org.elasticsearch:elasticsearch-spark-13_2.11:5.3.1` depending on your scala language version (2.10.x vs 2.11.x)

---

<div class="post-metadata">

**Author:** ![David\_Latham](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/david_latham/32/21473_2.png) [@David\_Latham](https://discuss.elastic.co/u/David_Latham)\
**Post date:** [September 6, 2017, 2:00pm UTC](https://discuss.elastic.co/t/pyspark-dataframe-save-error/98277/3 "2017-09-06T14:00:45Z")

</div>

running spark 2.10.4 so downloaded the first one. Still no love. Question: not sure about how to use artifacts since i added the .jar to my --jars list in my spark startup script (added it along with the hadoop-5.3.1 jar then by itself. Is this correct or am i doing something wrong? both ways produce the same error. thanks

---

<div class="post-metadata">

**Author:** ![David\_Latham](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/david_latham/32/21473_2.png) [@David\_Latham](https://discuss.elastic.co/u/David_Latham)\
**Post date:** [September 7, 2017, 7:19pm UTC](https://discuss.elastic.co/t/pyspark-dataframe-save-error/98277/4 "2017-09-07T19:19:01Z")

</div>

update: tested with scala and produced the same error.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 5, 2017, 7:19pm UTC](https://discuss.elastic.co/t/pyspark-dataframe-save-error/98277/5 "2017-10-05T19:19:53Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.