Class org.elasticsearch.spark.sql.SparkSQLCompatibilityLevel not found

Playing with es4Hadoop 5.2.1 and Spark, I built a Java application with Maven deps:

<dependencies>
    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-core_2.10</artifactId>
        <version>1.6.2</version>
    </dependency>

    <!-- https://mvnrepository.com/artifact/org.apache.spark/spark-sql_2.10 -->
    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-sql_2.10</artifactId>
        <version>1.6.2</version>
    </dependency>

    <!-- https://mvnrepository.com/artifact/org.elasticsearch/elasticsearch-hadoop -->
    <dependency>
        <groupId>org.elasticsearch</groupId>
        <artifactId>elasticsearch-spark-13_2.10</artifactId>
        <version>5.2.1</version>
    </dependency>

</dependencies>

Then when I run it (spark-submit --master local) against spark 1.6.2 on a MacBook:

Caused by: org.elasticsearch.hadoop.EsHadoopIllegalStateException: Cannot load class [org.elasticsearch.spark.sql.SparkSQLCompatibilityLevel]

  • ES library is built inside the Jar
  • The same project works fine with elasticsearch-hadoop:2.4.4 instead

@ebuildy This class was added during the 5.0 release. Do any other versions of 5.x malfunction for you, or just this one?

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.