# Issue Using the Connector from PySpark in 7.17.3 **URL:** \ **Category:** Elasticsearch\ **Tags:** es-hadoop\ **Created:** [May 6, 2022, 4:52am UTC](https://discuss.elastic.co/t/issue-using-the-connector-from-pyspark-in-7-17-3/304084 "2022-05-06T04:52:52Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![ito](https://avatars.discourse-cdn.com/v4/letter/i/ec9cab/32.png) [@ito](https://discuss.elastic.co/u/ito)\ **Post date:** [May 6, 2022, 4:52am UTC](https://discuss.elastic.co/t/issue-using-the-connector-from-pyspark-in-7-17-3/304084/1 "2022-05-06T04:52:52Z")
I have Elasticsearch 7.17.3 running on localhost:9200, so I downloaded [Elasticsearch for Apache Hadoop 7.17.3](https://www.elastic.co/downloads/past-releases/elasticsearch-apache-hadoop-7-17-3). Then I copied Elasticsearch-spark-20\_2.11-7.17.3.jar to my jar folder. I read "[Using the connector from PySpark](https://www.elastic.co/guide/en/elasticsearch/hadoop/7.17/spark.html#spark-python)" and was able to read data from Elasticsearch by adapting the first snippet with newAPIHadoopRDD as shown below. ```auto conf = { "es.resource" : "myindex" } es_rdd = sc.newAPIHadoopRDD(inputFormatClass="org.elasticsearch.hadoop.mr.EsInputFormat", keyClass="org.apache.hadoop.io.NullWritable", valueClass="org.elasticsearch.hadoop.mr.LinkedMapWritable", conf=conf) es_rdd.first() ``` But when I follow the second snippet with `org.elasticsearch.spark.sql` and run the code shown below, I get the following NoClassDefFoundError error: ```auto sqlContext = SQLContext(sc) df = sqlContext.read.format("org.elasticsearch.spark.sql").load("myindex") df.printSchema() ``` ```auto Py4JJavaError: An error occurred while calling o88.load. : java.lang.NoClassDefFoundError: scala/Product$class at org.elasticsearch.spark.sql.ElasticsearchRelation.(DefaultSource.scala:221) at org.elasticsearch.spark.sql.DefaultSource.createRelation(DefaultSource.scala:97) at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:350) at org.apache.spark.sql.DataFrameReader.loadV1Source(DataFrameReader.scala:274) at org.apache.spark.sql.DataFrameReader.$anonfun$load$3(DataFrameReader.scala:245) at scala.Option.getOrElse(Option.scala:189) at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:245) at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:188) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244) at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357) at py4j.Gateway.invoke(Gateway.java:282) at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132) at py4j.commands.CallCommand.execute(CallCommand.java:79) at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:182) at py4j.ClientServerConnection.run(ClientServerConnection.java:106) at java.lang.Thread.run(Thread.java:748) ``` What am I doing wrong? For your information, I am using PySpark 3.2.1. ---
**Author:** ![Keith\_Massey](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/keith_massey/32/83666_2.png) [@Keith\_Massey](https://discuss.elastic.co/u/Keith_Massey)\ **Post date:** [May 6, 2022, 10:18pm UTC](https://discuss.elastic.co/t/issue-using-the-connector-from-pyspark-in-7-17-3/304084/2 "2022-05-06T22:18:50Z")
This looks similar to [Elasticsearch-hadoop Scala version update to 2.12 · Issue #1949 · elastic/elasticsearch-hadoop · GitHub](https://github.com/elastic/elasticsearch-hadoop/issues/1949). We don't make it clear enough in our documentation yet, but the Elasticsearch-spark jar in the big Elasticsearch-hadoop artifact only works spark 2.x with scala 2.11. In that spark jar name (Elasticsearch-spark-20\_2.11-7.17.3.jar), the first number (20) is code for spark version 2.x. The second number (2.11) is the scala version. The third number (7.17.3) is the Elasticsearch version. Unfortunately neither scala nor spark tend to be compatible across versions. So for spark 3.2.1 you'll want [elasticsearch-spark-30\_2.12](https://search.maven.org/artifact/org.elasticsearch/elasticsearch-spark-30_2.12/7.17.3/jar). You'll have to use scala 2.12 if you are on 7.17.3. Scala 2.13 was only added in es-hadoop 8.1.0. ---
**Author:** ![Keith\_Massey](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/keith_massey/32/83666_2.png) [@Keith\_Massey](https://discuss.elastic.co/u/Keith_Massey)\ **Post date:** [May 6, 2022, 10:36pm UTC](https://discuss.elastic.co/t/issue-using-the-connector-from-pyspark-in-7-17-3/304084/3 "2022-05-06T22:36:30Z")
This is a little messy to try to capture on this site, but here's a view of which versions of scala and spark work with each es-spark artifact in the latest version of es-hadoop (8.2). The main differences with 7.17 are that scala 2.13 does not work on 7.17 and (I believe) spark 3.2.x is not supported on 7.17. | **es-hadoop 8.2** | | Scala Version | | | | 2.10 | 2.11 | 2.12 | 2.13 | 3.0 | | | 1.3-1.6 | elasticsearch-spark-13\_2.10 | elasticsearch-spark-13\_2.11 | X | X | X | | Spark Version | 2.x | elasticsearch-spark-20\_2.10 | elasticsearch-spark-20\_2.11 | elasticsearch-spark-20\_2.12 | X | X | | | 3.0-3.1 | X | X | elasticsearch-spark-30\_2.12 | X | X | | | 3.2 | X | X | elasticsearch-spark-30\_2.12 | elasticsearch-spark-30\_2.13 | X | ---
**Author:** ![ito](https://avatars.discourse-cdn.com/v4/letter/i/ec9cab/32.png) [@ito](https://discuss.elastic.co/u/ito)\ **Post date:** [May 7, 2022, 12:18am UTC](https://discuss.elastic.co/t/issue-using-the-connector-from-pyspark-in-7-17-3/304084/4 "2022-05-07T00:18:45Z")
Thanks @Keith_Massey. [elasticsearch-spark-30\_2.12](https://search.maven.org/artifact/org.elasticsearch/elasticsearch-spark-30_2.12/7.17.3/jar) works with PySpark 3.2.1 and Elasticsearch 7.17.3. I am new to Spark, and it did not occur to me that I had to check the version of Scala as I only use Python and PySpark. I should have paid attention to the version compatibility matrix on the [Elasticsearch for Apache Hadoop 7.17 installation page](https://www.elastic.co/guide/en/elasticsearch/hadoop/7.17/install.html) that shows the following: | Spark Version | Scala Version | ES-Hadoop Artifact ID | | --- | --- | --- | | 2.0+ | 2.11 | `elasticsearch-spark-20_2.11` | | 2.0+ | 2.12 | `elasticsearch-spark-20_2.12` | | 3.0+ | 2.12 | `elasticsearch-spark-30_2.12` | It now makes sense that `elasticsearch-spark-20_2.11-7.17.3.jar` in the downloaded zip file does not work. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [June 4, 2022, 12:19am UTC](https://discuss.elastic.co/t/issue-using-the-connector-from-pyspark-in-7-17-3/304084/5 "2022-06-04T00:19:04Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.