Unable to insert data into ES through spark-submit - works with pyspark

(Rohitgcs) #1

This is the line I use:
.write.options(**es_write_conf).mode("append").format("org.elasticsearch.spark.sql").save(esPath)

When I run it in a shell launching it with pyspark -jars /usr/lib/spark/jars/elasticsearch-hadoop-5.6.9.jar it runs perfectly. But when I tried to run it through spark-submit -jars /usr/lib/spark/jars/elasticsearch-hadoop-5.6.9.jar I get a ClassNotFoundException.

What would I need to do to figure out why it doesn't work on spark-submit?

(James Baiera) #2

Can you add the full ClassNotFoundException text here?

(Rohitgcs) #3

Error output here in the gist: