# Not able to connect to elastic from pyspark application

**URL:** <https://discuss.elastic.co/t/not-able-to-connect-to-elastic-from-pyspark-application/375005>\
**Category:** Elasticsearch\
**Tags:** docker\
**Created:** [February 25, 2025, 10:44am UTC](https://discuss.elastic.co/t/not-able-to-connect-to-elastic-from-pyspark-application/375005 "2025-02-25T10:44:34Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![meghananagaraja](https://avatars.discourse-cdn.com/v4/letter/m/8e7dd6/32.png) [@meghananagaraja](https://discuss.elastic.co/u/meghananagaraja)\
**Post date:** [February 25, 2025, 10:44am UTC](https://discuss.elastic.co/t/not-able-to-connect-to-elastic-from-pyspark-application/375005/1 "2025-02-25T10:44:34Z")

</div>

Hi All,  
I am running Kafka, spark and elastic separately on docker containers. I use a python pyspark application to connect with elastic. But i am getting an error

```auto
spark-job-2 | INFO:elastic_transport.transport:HEAD https://elasticsearch:9200/ [status:N/A duration:0.005s]
spark-job-2 | WARNING:elastic_transport.node_pool:Node <Urllib3HttpNode(https://elasticsearch:9200)> has failed for 3 times in a row, putting on 4 second timeout
spark-job-2 | WARNING:elastic_transport.transport:Retrying request after failure (attempt 2 of 3)
spark-job-2 | Traceback (most recent call last):
spark-job-2 | File "/path/to/python3.9/site-packages/elastic_transport/_transport.py", line 328, in perform_request
spark-job-2 | meta, raw_data = node.perform_request(
spark-job-2 | File "/usr/local/lib/python3.9/site-packages/elastic_transport/_node/_http_urllib3.py", line 202, in perform_request
spark-job-2 | raise err from None
spark-job-2 | elastic_transport.ConnectionError: Connection error caused by: NewConnectionError(<urllib3.connection.HTTPSConnection object at 0x7f2b0e79b790>: Failed to establish a new connection: [Errno -2] Name or service not known)
spark-job-2 | INFO:elastic_transport.transport:HEAD https://elasticsearch:9200/ [status:N/A duration:0.006s]
spark-job-2 | WARNING:elastic_transport.node_pool:Node <Urllib3HttpNode(https://elasticsearch:9200)> has failed for 4 times in a row, putting on 8 second timeout
spark-job-2 | ERROR: __main__ :Not connected to Elasticsearch

```

this is some additional config from python script

```auto
def getSparkSessionInstance(sparkConf: SparkConf) -> SparkSession:
    '''Create a singleton SparkSession instance'''
    if 'sparkSessionSingletonInstance' not in globals():
        globals()['sparkSessionSingletonInstance'] = SparkSession\
            .builder\
            .config(conf=sparkConf)\
            .config("spark.es.nodes", " ****") \
            .config("spark.es.port", "9200") \
            .config("spark.es.nodes.wan.only", "true") \
            .config("spark.es.index.auto.create", "true") \
            .config("spark.es.scheme", "http") \
            .config("spark.es.net.http.auth.user", " ****") \
            .config("spark.es.net.http.auth.pass", " ****") \
            .getOrCreate()
    return globals()['sparkSessionSingletonInstance']

```

what am i doing incorrectly.

---

<div class="post-metadata">

**Author:** ![RainTown](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/raintown/32/140206_2.png) [@RainTown](https://discuss.elastic.co/u/RainTown)\
**Post date:** [February 25, 2025, 11:18am UTC](https://discuss.elastic.co/t/not-able-to-connect-to-elastic-from-pyspark-application/375005/2 "2025-02-25T11:18:42Z")

</div>

> [@meghananagaraja](#):
>
> `Urllib3HttpNode(https://elasticsearch:9200)> has failed for 4 times`

from wherever this is being executed, would the name "elasticsearch" resolve to the correct IP address?

> [@meghananagaraja](#):
>
> `[Errno -2] Name or service not known)`

I am suspecting no.

---

<div class="post-metadata">

**Author:** ![meghananagaraja](https://avatars.discourse-cdn.com/v4/letter/m/8e7dd6/32.png) [@meghananagaraja](https://discuss.elastic.co/u/meghananagaraja)\
**Post date:** [February 26, 2025, 8:51am UTC](https://discuss.elastic.co/t/not-able-to-connect-to-elastic-from-pyspark-application/375005/3 "2025-02-26T08:51:28Z")

</div>

Thankyou @RainTown. you were right!

---

<div class="post-metadata">

**Author:** ![RainTown](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/raintown/32/140206_2.png) [@RainTown](https://discuss.elastic.co/u/RainTown)\
**Post date:** [February 26, 2025, 9:06am UTC](https://discuss.elastic.co/t/not-able-to-connect-to-elastic-from-pyspark-application/375005/4 "2025-02-26T09:06:06Z")

</div>

Good to hear.

You can accept one of the answers to close the thread, and I wish you luck with your project!