Playing with es4Hadoop 5.2.1 and Spark, I built a Java application with Maven deps:
<dependencies>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.10</artifactId>
<version>1.6.2</version>
</dependency>
<!-- https://mvnrepository.com/artifact/org.apache.spark/spark-sql_2.10 -->
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_2.10</artifactId>
<version>1.6.2</version>
</dependency>
<!-- https://mvnrepository.com/artifact/org.elasticsearch/elasticsearch-hadoop -->
<dependency>
<groupId>org.elasticsearch</groupId>
<artifactId>elasticsearch-spark-13_2.10</artifactId>
<version>5.2.1</version>
</dependency>
</dependencies>
Then when I run it (spark-submit --master local) against spark 1.6.2 on a MacBook:
Caused by: org.elasticsearch.hadoop.EsHadoopIllegalStateException: Cannot load class [org.elasticsearch.spark.sql.SparkSQLCompatibilityLevel]
- ES library is built inside the Jar
- The same project works fine with elasticsearch-hadoop:2.4.4 instead