Class org.elasticsearch.spark.sql.SparkSQLCompatibilityLevel not found

Elastic Stack Elasticsearch
1

Playing with es4Hadoop 5.2.1 and Spark, I built a Java application with Maven deps:

<dependencies>
    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-core_2.10</artifactId>
        <version>1.6.2</version>
    </dependency>

    <!-- https://mvnrepository.com/artifact/org.apache.spark/spark-sql_2.10 -->
    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-sql_2.10</artifactId>
        <version>1.6.2</version>
    </dependency>

    <!-- https://mvnrepository.com/artifact/org.elasticsearch/elasticsearch-hadoop -->
    <dependency>
        <groupId>org.elasticsearch</groupId>
        <artifactId>elasticsearch-spark-13_2.10</artifactId>
        <version>5.2.1</version>
    </dependency>

</dependencies>

Then when I run it (spark-submit --master local) against spark 1.6.2 on a MacBook:

Caused by: org.elasticsearch.hadoop.EsHadoopIllegalStateException: Cannot load class [org.elasticsearch.spark.sql.SparkSQLCompatibilityLevel]

  • ES library is built inside the Jar
  • The same project works fine with elasticsearch-hadoop:2.4.4 instead
2

@ebuildy This class was added during the 5.0 release. Do any other versions of 5.x malfunction for you, or just this one?