This is the line I use: .write.options(**es_write_conf).mode("append").format("org.elasticsearch.spark.sql").save(esPath)
When I run it in a shell launching it with pyspark -jars /usr/lib/spark/jars/elasticsearch-hadoop-5.6.9.jar it runs perfectly. But when I tried to run it through spark-submit -jars /usr/lib/spark/jars/elasticsearch-hadoop-5.6.9.jar I get a ClassNotFoundException.
What would I need to do to figure out why it doesn't work on spark-submit?
This seems like a pretty strange problem here. I would check to make sure that you are using the correct ES-Hadoop version for your distribution of Spark as well as for your distribution of Scala.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.