Pyspark Dataframe Save Error

[David_Latham]

i'm running spark 1.5 and used es-hadoop 2.4 to connect to ES 2.1.1 server. reads and writes worked fine. after upgrading ES to 5.3.1 and downloading es-hadoop 5.3.1 (spark is still at 1.5) the reads work fine, but when attempting to write i get an error:

DF = sqlContext.read.parquet('data/on_time_performance.parquet')

DF.write.format("org.elasticsearch.spark.sql")
.option("es.resource","agile_data_science/on_time_performance")
.mode("overwrite")
.save()

Py4JJavaError: An error occurred while calling o39.save.
: java.lang.AbstractMethodError: org.elasticsearch.spark.sql.DefaultSource.createRelation(Lorg/apache/spark/sql/SQLContext;Lorg/apache/spark/sql/SaveMode;Lscala/collection/immutable/Map;Lorg/apache/spark/sql/Dataset;)Lorg/apache/spark/sql/sources/BaseRelation;
at org.apache.spark.sql.execution.datasources.DataSource.write(DataSource.scala:429)
at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:211)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:237)
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
at py4j.Gateway.invoke(Gateway.java:280)
at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:128)
at py4j.commands.CallCommand.execute(CallCommand.java:79)
at py4j.GatewayConnection.run(GatewayConnection.java:211)
at java.lang.Thread.run(Thread.java:745)

all the threads say to use the right driver, but which one? i've got about 10 downloaded from 2.1 - > 5.3.1. none of the 2.X ones will work against 5.3.1 ES server, and all the 5.X drivers give the same error.

Thanks,

[james.baiera]

Since you're using spark 1.5 you need the 1.3-1.6 compatibility artifact for ES-Hadoop 5.3.1. Try org.elasticsearch:elasticsearch-spark-13_2.10:5.3.1 or org.elasticsearch:elasticsearch-spark-13_2.11:5.3.1 depending on your scala language version (2.10.x vs 2.11.x)

[David_Latham]

running spark 2.10.4 so downloaded the first one. Still no love. Question: not sure about how to use artifacts since i added the .jar to my --jars list in my spark startup script (added it along with the hadoop-5.3.1 jar then by itself. Is this correct or am i doing something wrong? both ways produce the same error. thanks

[David_Latham]

update: tested with scala and produced the same error.