No able to write in Elasticsearch index using Spark 2.0.0


(Sumit Gupta) #1

Hi All,

Here is my setup information
ES version: 2.3.3
Spark: 2.0.0
ES-Hadoop jar: elasticsearch-hadoop-2.3.3.jar
ES-Spark Jar: elasticsearch-spark_2.11-2.3.3.jar

I am trying to read data from ES index and trying to perform join using Spark using and again trying to insert in Elasticsearch index.Reading data from ES is fine and join is also fine but when i am trying to insert response (basically something like this "df.write.format("org.elasticsearch.spark.sql").option("es.resource", "combinejoin/join").save()" )
it is throwing error:

Here is my full Stacktrace:
py4j.protocol.Py4JJavaError: An error occurred while calling o34.save.
: java.lang.AbstractMethodError: org.elasticsearch.spark.sql.DefaultSource.createRelation(Lorg/apache/spark/sql/SQLContext;Lorg/apache/spark/sql/SaveMode;Lscala/collection/immutable/Map;Lorg/apache/spark/sql/Dataset;)Lorg/apache/spark/sql/sources/BaseRelation;
at org.apache.spark.sql.execution.datasources.DataSource.write(DataSource.scala:429)
at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:211)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:237)
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
at py4j.Gateway.invoke(Gateway.java:280)
at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:128)
at py4j.commands.CallCommand.execute(CallCommand.java:79)
at py4j.GatewayConnection.run(GatewayConnection.java:211)
at java.lang.Thread.run(Thread.java:745)


(James Baiera) #2

Spark 2.0 is only supported in ES Hadoop 5.0.0-alpha5 and above.
https://mvnrepository.com/artifact/org.elasticsearch/elasticsearch-spark-20_2.10
https://mvnrepository.com/artifact/org.elasticsearch/elasticsearch-spark-20_2.11


(system) #3