# Error while running pyspark connecting to elastic search

**URL:** <https://discuss.elastic.co/t/error-while-running-pyspark-connecting-to-elastic-search/351195>\
**Category:** Elasticsearch\
**Tags:** es-hadoop\
**Created:** [January 16, 2024, 6:42pm UTC](https://discuss.elastic.co/t/error-while-running-pyspark-connecting-to-elastic-search/351195 "2024-01-16T18:42:38Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![kashi\_mn](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kashi_mn/32/19639_2.png) [@kashi\_mn](https://discuss.elastic.co/u/kashi_mn)\
**Post date:** [January 16, 2024, 6:42pm UTC](https://discuss.elastic.co/t/error-while-running-pyspark-connecting-to-elastic-search/351195/1 "2024-01-16T18:42:38Z")

</div>

I am new to Pyspark and currently running pyspark and trying to connect to elasticsearch running on localhost.

Below are the details :

Elastic Details : localhost  
Port : 9200  
No https.

**Code :**

```auto
from pyspark.sql import SparkSession
spark = SparkSession.builder.getOrCreate()
df = spark.read.format("org.elasticsearch.spark.sql").option("es.nodes", "localhost").option("es.port", "9200").load("products")
df.show()

```

**Command running via terminal :**

```auto
spark-submit --packages org.apache.spark:spark-sql-kafka-0-10_2.12:3.2.0 PySp.py

```

**pyspark --version:**  
version 3.2.0  
Using Scala version 2.12.15, OpenJDK 64-Bit Server VM, 11.0.21

**Error :**

```auto
df = spark.read.format("org.elasticsearch.spark.sql").option("es.nodes", "localhost").option("es.port", "9200").load("products")
  File "/opt/spark/python/lib/pyspark.zip/pyspark/sql/readwriter.py", line 158, in load
  File "/opt/spark/python/lib/py4j-0.10.9.2-src.zip/py4j/java_gateway.py", line 1309, in __call__
  File "/opt/spark/python/lib/pyspark.zip/pyspark/sql/utils.py", line 111, in deco
  File "/opt/spark/python/lib/py4j-0.10.9.2-src.zip/py4j/protocol.py", line 326, in get_return_value
py4j.protocol.Py4JJavaError: An error occurred while calling o28.load.
: java.util.ServiceConfigurationError: org.apache.spark.sql.sources.DataSourceRegister: org.elasticsearch.spark.sql.DefaultSource15 Unable to get public no-arg constructor
	at java.base/java.util.ServiceLoader.fail(ServiceLoader.java:582)
	at java.base/java.util.ServiceLoader.getConstructor(ServiceLoader.java:673)
	at java.base/java.util.ServiceLoader$LazyClassPathLookupIterator.hasNextService(ServiceLoader.java:1233)
	at java.base/java.util.ServiceLoader$LazyClassPathLookupIterator.hasNext(ServiceLoader.java:1265)
	at java.base/java.util.ServiceLoader$2.hasNext(ServiceLoader.java:1300)
	at java.base/java.util.ServiceLoader$3.hasNext(ServiceLoader.java:1385)
	at scala.collection.convert.Wrappers$JIteratorWrapper.hasNext(Wrappers.scala:45)

```

Please can you help me how to resolve this issue.

---

<div class="post-metadata">

**Author:** ![Keith\_Massey](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/keith_massey/32/83666_2.png) [@Keith\_Massey](https://discuss.elastic.co/u/Keith_Massey)\
**Post date:** [January 22, 2024, 3:53pm UTC](https://discuss.elastic.co/t/error-while-running-pyspark-connecting-to-elastic-search/351195/2 "2024-01-22T15:53:46Z")

</div>

Hi @kashi_mn. This looks like the same error we saw at [Pyspark-Elasticsearch connectivity and latest version compatibilty - #6 by Bramha](https://discuss.elastic.co/t/pyspark-elasticsearch-connectivity-and-latest-version-compatibilty/325289/6). It's worth double-checking which es-spark jar you're giving it.

---

<div class="post-metadata">

**Author:** ![kashi\_mn](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kashi_mn/32/19639_2.png) [@kashi\_mn](https://discuss.elastic.co/u/kashi_mn)\
**Post date:** [January 24, 2024, 5:03am UTC](https://discuss.elastic.co/t/error-while-running-pyspark-connecting-to-elastic-search/351195/3 "2024-01-24T05:03:36Z")

</div>

Hi @Keith_Massey,  
Thanks for the response. I am running elastic instance of 8.11.1 version.  
I see that you have already recommended on the mentioned thread about the spark, elastic-spark connector versions. Will try the same.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [February 21, 2024, 5:04am UTC](https://discuss.elastic.co/t/error-while-running-pyspark-connecting-to-elastic-search/351195/4 "2024-02-21T05:04:00Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.