# PySpark - Seeing a 'ConnectionTimedOut' to a local ElasticSearch

**URL:** <https://discuss.elastic.co/t/pyspark-seeing-a-connectiontimedout-to-a-local-elasticsearch/153879>\
**Category:** Elasticsearch\
**Tags:** es-hadoop\
**Created:** [October 24, 2018, 9:59pm UTC](https://discuss.elastic.co/t/pyspark-seeing-a-connectiontimedout-to-a-local-elasticsearch/153879 "2018-10-24T21:59:18Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![initialv](https://avatars.discourse-cdn.com/v4/letter/i/4da419/32.png) [@initialv](https://discuss.elastic.co/u/initialv)\
**Post date:** [October 24, 2018, 9:59pm UTC](https://discuss.elastic.co/t/pyspark-seeing-a-connectiontimedout-to-a-local-elasticsearch/153879/1 "2018-10-24T21:59:18Z")

</div>

I have a simple elasticsearch query that I want to receive that count of. On my local mac environment, the below setup works as expected.

On an Ubuntu machine, I see `ERROR NetworkClient:144 - Node [10.0.0.1:9200] failed` (also odd, because the configuration points to 127.0.0.1).

Environment that does not work:  
OS: Ubuntu  
Python: 2.7.10  
Spark: 2.3.1 and 2.3.2  
ElasticSearch: 5.6.5  
ES Hadoop Jar: elasticsearch-spark-20\_2.11-5.6.5.jar

Environment that does work:  
OS: mac  
Python: 2.7.10  
Spark: 2.2.1, 2.3.1, and 2.3.2  
ElasticSearch: 5.6.3  
ES Hadoop Jar: elasticsearch-spark-20\_2.11-5.6.3.jar

PySpark is run from the interactive shell via

```bash
path/to/pyspark \
   --jars /path/to/elasticsearch-spark-20_2.11-5.6.5.jar 
   --master local

```

The code I run is as follows:

```python
# I put the timeout to 10 seconds to see if I could get the error quicker, but it appears to still wait the full 1 minute (which is the default as per the docs)
config = {
   'es.nodes': '127.0.0.1:9200', 
   'es.scroll.size': '9000', 
   'es.resource': 'mydata/data', 
   'es.query': '{"query": {"match_all": {}}}', 
   'es.http.timeout': '10s'}
dataframe = sqlContext.read.format("org.elasticsearch.spark.sql").options(**config).load()
dataframe.count() 
# waits for about a minute then shows the below error, then retries with 127.0.0.1 and errors out again
# ERROR NetworkClient:144 - Node [10.0.0.1:9200] failed (Connection timed out (Connection timed out)); selected next node [127.0.0.1:9200]

```

---

<div class="post-metadata">

**Author:** ![initialv](https://avatars.discourse-cdn.com/v4/letter/i/4da419/32.png) [@initialv](https://discuss.elastic.co/u/initialv)\
**Post date:** [October 25, 2018, 5:04pm UTC](https://discuss.elastic.co/t/pyspark-seeing-a-connectiontimedout-to-a-local-elasticsearch/153879/2 "2018-10-25T17:04:48Z")

</div>

Found the issue. The elasticsearch.yml configuration had a line as follows:

```auto
discovery.type: single-node

```

Removing this line allowed spark to connect to the elasticsearch local node. I'm not sure why this caused the issue, so any insight on that would be appreciated.

---

<div class="post-metadata">

**Author:** ![james.baiera](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/james.baiera/32/10209_2.png) [@james.baiera](https://discuss.elastic.co/u/james.baiera)\
**Post date:** [October 25, 2018, 8:36pm UTC](https://discuss.elastic.co/t/pyspark-seeing-a-connectiontimedout-to-a-local-elasticsearch/153879/3 "2018-10-25T20:36:51Z")

</div>

if you have the concrete logs from the run available, they might help shed some light on the issue. It's possible that the local node that you are connecting to picked up the 10.0.0.1 address as its publish address, and so the connector tried communicating with the node through that address after it performed its node discovery process.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [November 22, 2018, 8:36pm UTC](https://discuss.elastic.co/t/pyspark-seeing-a-connectiontimedout-to-a-local-elasticsearch/153879/4 "2018-11-22T20:36:52Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.