Unable to insert data into ES through spark-submit - works with pyspark

Elastic Stack Elasticsearch
1

This is the line I use:
.write.options(**es_write_conf).mode("append").format("org.elasticsearch.spark.sql").save(esPath)

When I run it in a shell launching it with pyspark -jars /usr/lib/spark/jars/elasticsearch-hadoop-5.6.9.jar it runs perfectly. But when I tried to run it through spark-submit -jars /usr/lib/spark/jars/elasticsearch-hadoop-5.6.9.jar I get a ClassNotFoundException.

What would I need to do to figure out why it doesn't work on spark-submit?

2

Can you add the full ClassNotFoundException text here?

3

Error output here in the gist:

4

@james.baiera any idea why this would happen?

5

This seems like a pretty strange problem here. I would check to make sure that you are using the correct ES-Hadoop version for your distribution of Spark as well as for your distribution of Scala.