Cannot load class org.elasticsearch.spark.sql.SparkSQLCompatibilityLevel

[SKing]

I am running a Java Spark application. I can run the application from inside the intellij IDE. However, if I try to run it using the application built as a jar I get the following error.

Caused by: org.elasticsearch.hadoop.EsHadoopIllegalStateException: Cannot load class [org.elasticsearch.spark.sql.SparkSQLCompatibilityLevel]

Using
spark-core_2.10 v 1.6.3
elasticsearch-spark-13_2.10 v 5.6.8

The problem seems identical to the one encountered in this thread.

Class org.elasticsearch.spark.sql.SparkSQLCompatibilityLevel not found

Playing with es4Hadoop 5.2.1 and Spark, I built a Java application with Maven deps: <dependencies> <dependency> <groupId>org.apache.spark</groupId> <artifactId>spark-core_2.10</artifactId> <version>1.6.2</version> </dependency> <!-- https://mvnrepository.com/artifact/org.apache.spark/spark-sql_2.10 --> <dependency> <groupId>org.apache.spark</groupId> <artifactId>spark-sql_2.10</artifactId> <version>1.6.2</version> </dependency>…

However, no mention is given of how to fix this.

[james.baiera]

Since the job succeeds when launched from the IDE but not when built as a jar, I would double check that the ES-Spark jars are indeed available on the classpath for your spark program.

[ebuildy (Thomas Decaux)]

Hello,

The cause is the class is loaded dynamically, so this can bug if you optimize the jar (or minimize it via Maven share plugin for instance).