Unable to insert data into ES through spark-submit - works with pyspark

This is the line I use:
.write.options(**es_write_conf).mode("append").format("org.elasticsearch.spark.sql").save(esPath)

When I run it in a shell launching it with pyspark -jars /usr/lib/spark/jars/elasticsearch-hadoop-5.6.9.jar it runs perfectly. But when I tried to run it through spark-submit -jars /usr/lib/spark/jars/elasticsearch-hadoop-5.6.9.jar I get a ClassNotFoundException.

What would I need to do to figure out why it doesn't work on spark-submit?

Can you add the full ClassNotFoundException text here?

Error output here in the gist:

@james.baiera any idea why this would happen?

This seems like a pretty strange problem here. I would check to make sure that you are using the correct ES-Hadoop version for your distribution of Spark as well as for your distribution of Scala.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.