# Error in Databricks 7.6 and Elastic 7.1.1 \*\*java.lang.ClassNotFoundException: Failed to find data source: org.elasticsearch.spark.sql.\*\* **URL:** \ **Category:** Elasticsearch\ **Tags:** es-hadoop\ **Created:** [May 27, 2021, 11:01pm UTC](https://discuss.elastic.co/t/error-in-databricks-7-6-and-elastic-7-1-1-java-lang-classnotfoundexception-failed-to-find-data-source-org-elasticsearch-spark-sql/274272 "2021-05-27T23:01:58Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![mlo](https://avatars.discourse-cdn.com/v4/letter/m/7c8e57/32.png) [@mlo](https://discuss.elastic.co/u/mlo)\ **Post date:** [May 27, 2021, 11:01pm UTC](https://discuss.elastic.co/t/error-in-databricks-7-6-and-elastic-7-1-1-java-lang-classnotfoundexception-failed-to-find-data-source-org-elasticsearch-spark-sql/274272/1 "2021-05-27T23:01:58Z")
We are running Databricks 7.6 (includes Apache Spark 3.0.1, Scala 2.12) and istalled via **jip install org.elasticsearch:elasticsearch-spark-30\_2.12:7.12.1** and now I am getting this error **java.lang.ClassNotFoundException: Failed to find data source: org.elasticsearch.spark.sql.** ---
**Author:** ![Matthew\_Isett](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/matthew_isett/32/57770_2.png) [@Matthew\_Isett](https://discuss.elastic.co/u/Matthew_Isett)\ **Post date:** [June 4, 2021, 1:46pm UTC](https://discuss.elastic.co/t/error-in-databricks-7-6-and-elastic-7-1-1-java-lang-classnotfoundexception-failed-to-find-data-source-org-elasticsearch-spark-sql/274272/2 "2021-06-04T13:46:38Z")
@mlo Can you run a test that utilizes Spark 3.0 classes outside of spark.sql? ``` > import org.elasticsearch.spark._ > val conf = ... > val sc = new SparkContext(conf) > sc.esRDD("radio/artists", "?q=me*") ``` Also, post your class here for review AND > jip list > java -version And elasticsearch version ( you posted here 7.1?) ls on the elasticsearch home dir > ls ---
**Author:** ![gotigers](https://avatars.discourse-cdn.com/v4/letter/g/bbe5ce/32.png) [@gotigers](https://discuss.elastic.co/u/gotigers)\ **Post date:** [June 14, 2021, 4:23pm UTC](https://discuss.elastic.co/t/error-in-databricks-7-6-and-elastic-7-1-1-java-lang-classnotfoundexception-failed-to-find-data-source-org-elasticsearch-spark-sql/274272/3 "2021-06-14T16:23:06Z")
Hey Matt. Using Elastic 7.11 Also I ran this as you suggested and it seemed to work without error. %scala import org.apache.spark.SparkConf import org.elasticsearch.spark.\_ val sc = SparkContext.getOrCreate() sc.esRDD("radio/artists", "?q=me\*") ​ **Result:** import org.apache.spark.SparkConf import org.elasticsearch.spark.\_ sc: org.apache.spark.SparkContext = org.apache.spark.SparkContext@6b1cbda7 res2: org.apache.spark.rdd.RDD[(String, scala.collection.Map[String,AnyRef])] = ScalaEsRDD[44] at RDD at AbstractEsRDD.scala:41 ---
**Author:** ![gotigers](https://avatars.discourse-cdn.com/v4/letter/g/bbe5ce/32.png) [@gotigers](https://discuss.elastic.co/u/gotigers)\ **Post date:** [June 14, 2021, 4:24pm UTC](https://discuss.elastic.co/t/error-in-databricks-7-6-and-elastic-7-1-1-java-lang-classnotfoundexception-failed-to-find-data-source-org-elasticsearch-spark-sql/274272/4 "2021-06-14T16:24:49Z")
> [@Matthew\_Isett](#): > > jip list Also %sh java -version openjdk version "1.8.0\_282" OpenJDK Runtime Environment (Zulu 8.52.0.23-CA-linux64) (build 1.8.0\_282-b08) OpenJDK 64-Bit Server VM (Zulu 8.52.0.23-CA-linux64) (build 25.282-b08, mixed mode) ---
**Author:** ![gotigers](https://avatars.discourse-cdn.com/v4/letter/g/bbe5ce/32.png) [@gotigers](https://discuss.elastic.co/u/gotigers)\ **Post date:** [June 14, 2021, 4:26pm UTC](https://discuss.elastic.co/t/error-in-databricks-7-6-and-elastic-7-1-1-java-lang-classnotfoundexception-failed-to-find-data-source-org-elasticsearch-spark-sql/274272/5 "2021-06-14T16:26:30Z")
jip list returns nothing ---
**Author:** ![Matthew\_Isett](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/matthew_isett/32/57770_2.png) [@Matthew\_Isett](https://discuss.elastic.co/u/Matthew_Isett)\ **Post date:** [June 22, 2021, 1:12pm UTC](https://discuss.elastic.co/t/error-in-databricks-7-6-and-elastic-7-1-1-java-lang-classnotfoundexception-failed-to-find-data-source-org-elasticsearch-spark-sql/274272/6 "2021-06-22T13:12:23Z")
I put together the project and went with Java, instead of Scala - again just testing out different environments - and I got the import org.elasticsearch.spark.sql.api.java.JavaEsSparkSQL; -- class to execute correctly. POM.xml ```auto org.elasticsearch elasticsearch-spark-30_2.12 7.13.2 org.apache.spark spark-sql_2.12 3.0.2 ``` Either the packaged jar needs to contain both spark sql\*.jar and elasticsearch-spark\*.jar or install both jars on all clients that are executing the class. Class -- ```auto package hello; import java.util.Map; import org.apache.commons.lang.StringUtils; import org.joda.time.LocalTime; import org.apache.spark.api.java.JavaSparkContext; import org.apache.spark.api.java.JavaRDD; import org.apache.spark.SparkConf; //import org.apache.spark.sql.api.java.JavaSQLContext; import org.apache.spark.sql.SQLContext; import org.apache.spark.sql.Dataset; import org.apache.spark.sql.Row; import static org.elasticsearch.spark.rdd.api.java.JavaEsSpark.*; import org.elasticsearch.spark.sql.api.java.JavaEsSparkSQL; public class HelloWorld { public static void main(String[] args) { SparkConf conf = new SparkConf().setAppName("AppName").setMaster("local"); conf.set("es.index.auto.create", "true"); conf.set("es.nodes.wan.only", "true"); conf.set("es.nodes", "https:// ******.eastus2.azure.elastic-cloud.com"); conf.set("es.port", "9243"); conf.set("es.net.http.auth.user", " ********"); conf.set("es.net.http.auth.pass", " *******"); JavaSparkContext jsc = new JavaSparkContext(conf); // JavaRDD> esRDD = // esRDD(jsc, "filebeat-*/_doc", "?q=test").values(); // System.out.println(StringUtils.join(esRDD.collect(), ",")); SQLContext sql = new SQLContext(jsc); //test-message is a very simple index on my cloud cluster Dataset people = JavaEsSparkSQL.esDF(sql, "test-message"); System.out.println("String: " + people.first().toString()); } } ``` This is working in local mode. I noticed - or got hung up on DataFrame moved to Dataset - but that was spark-related not classpath related to org.elasticsearch.spark.sql. Spark psuedo code --\> ```auto import org.apache.spark.sql.SQLContext import org.elasticsearch.spark.sql._ val conf = (example in java all the things to set...) val sc = new SparkContext(conf) val sql = new SQLContext(sc) val people = sql.esDF("spark/people") ``` ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 20, 2021, 1:13pm UTC](https://discuss.elastic.co/t/error-in-databricks-7-6-and-elastic-7-1-1-java-lang-classnotfoundexception-failed-to-find-data-source-org-elasticsearch-spark-sql/274272/7 "2021-07-20T13:13:17Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.