Pyspark Dataframe Save Error

(David Latham) #1

i'm running spark 1.5 and used es-hadoop 2.4 to connect to ES 2.1.1 server. reads and writes worked fine. after upgrading ES to 5.3.1 and downloading es-hadoop 5.3.1 (spark is still at 1.5) the reads work fine, but when attempting to write i get an error:

DF ='data/on_time_performance.parquet')


Py4JJavaError: An error occurred while calling
: java.lang.AbstractMethodError: org.elasticsearch.spark.sql.DefaultSource.createRelation(Lorg/apache/spark/sql/SQLContext;Lorg/apache/spark/sql/SaveMode;Lscala/collection/immutable/Map;Lorg/apache/spark/sql/Dataset;)Lorg/apache/spark/sql/sources/BaseRelation;
at org.apache.spark.sql.execution.datasources.DataSource.write(DataSource.scala:429)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(
at sun.reflect.DelegatingMethodAccessorImpl.invoke(
at java.lang.reflect.Method.invoke(
at py4j.reflection.MethodInvoker.invoke(
at py4j.reflection.ReflectionEngine.invoke(
at py4j.Gateway.invoke(
at py4j.commands.AbstractCommand.invokeMethod(
at py4j.commands.CallCommand.execute(

all the threads say to use the right driver, but which one? i've got about 10 downloaded from 2.1 - > 5.3.1. none of the 2.X ones will work against 5.3.1 ES server, and all the 5.X drivers give the same error.


(James Baiera) #2

Since you're using spark 1.5 you need the 1.3-1.6 compatibility artifact for ES-Hadoop 5.3.1. Try org.elasticsearch:elasticsearch-spark-13_2.10:5.3.1 or org.elasticsearch:elasticsearch-spark-13_2.11:5.3.1 depending on your scala language version (2.10.x vs 2.11.x)

(David Latham) #3

running spark 2.10.4 so downloaded the first one. Still no love. Question: not sure about how to use artifacts since i added the .jar to my --jars list in my spark startup script (added it along with the hadoop-5.3.1 jar then by itself. Is this correct or am i doing something wrong? both ways produce the same error. thanks

(David Latham) #4

update: tested with scala and produced the same error.

(system) #5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.