Unable to insert data into ES through spark-submit - works with pyspark

(Rohitgcs) #1

This is the line I use:
.write.options(**es_write_conf).mode("append").format("org.elasticsearch.spark.sql").save(esPath)

When I run it in a shell launching it with pyspark -jars /usr/lib/spark/jars/elasticsearch-hadoop-5.6.9.jar it runs perfectly. But when I tried to run it through spark-submit -jars /usr/lib/spark/jars/elasticsearch-hadoop-5.6.9.jar I get a ClassNotFoundException.

What would I need to do to figure out why it doesn't work on spark-submit?

(James Baiera) #2

Can you add the full ClassNotFoundException text here?

(Rohitgcs) #3

Error output here in the gist:

(Rohitgcs) #4

@james.baiera any idea why this would happen?

(James Baiera) #5

This seems like a pretty strange problem here. I would check to make sure that you are using the correct ES-Hadoop version for your distribution of Spark as well as for your distribution of Scala.

(system) closed #6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.