Not able to connect to elastic from pyspark application

Elastic Stack Elasticsearch
1

Hi All,
I am running Kafka, spark and elastic separately on docker containers. I use a python pyspark application to connect with elastic. But i am getting an error

spark-job-2  | INFO:elastic_transport.transport:HEAD https://elasticsearch:9200/ [status:N/A duration:0.005s]
spark-job-2  | WARNING:elastic_transport.node_pool:Node <Urllib3HttpNode(https://elasticsearch:9200)> has failed for 3 times in a row, putting on 4 second timeout
spark-job-2  | WARNING:elastic_transport.transport:Retrying request after failure (attempt 2 of 3)
spark-job-2  | Traceback (most recent call last):
spark-job-2  |   File "/path/to/python3.9/site-packages/elastic_transport/_transport.py", line 328, in perform_request
spark-job-2  |     meta, raw_data = node.perform_request(
spark-job-2  |   File "/usr/local/lib/python3.9/site-packages/elastic_transport/_node/_http_urllib3.py", line 202, in perform_request
spark-job-2  |     raise err from None
spark-job-2  | elastic_transport.ConnectionError: Connection error caused by: NewConnectionError(<urllib3.connection.HTTPSConnection object at 0x7f2b0e79b790>: Failed to establish a new connection: [Errno -2] Name or service not known)
spark-job-2  | INFO:elastic_transport.transport:HEAD https://elasticsearch:9200/ [status:N/A duration:0.006s]
spark-job-2  | WARNING:elastic_transport.node_pool:Node <Urllib3HttpNode(https://elasticsearch:9200)> has failed for 4 times in a row, putting on 8 second timeout
spark-job-2  | ERROR:__main__:Not connected to Elasticsearch

this is some additional config from python script

def getSparkSessionInstance(sparkConf: SparkConf) -> SparkSession:
    '''Create a singleton SparkSession instance'''
    if 'sparkSessionSingletonInstance' not in globals():
        globals()['sparkSessionSingletonInstance'] = SparkSession\
            .builder\
            .config(conf=sparkConf)\
            .config("spark.es.nodes", "****") \
            .config("spark.es.port", "9200") \
            .config("spark.es.nodes.wan.only", "true") \
            .config("spark.es.index.auto.create", "true") \
            .config("spark.es.scheme", "http") \
            .config("spark.es.net.http.auth.user", "****") \
            .config("spark.es.net.http.auth.pass", "****") \
            .getOrCreate()
    return globals()['sparkSessionSingletonInstance']

what am i doing incorrectly.

2

from wherever this is being executed, would the name "elasticsearch" resolve to the correct IP address?

I am suspecting no.

3

Thankyou @RainTown. you were right!

1 Like
4

Good to hear.

You can accept one of the answers to close the thread, and I wish you luck with your project!

1 Like