# Connector for Elastic Search 8.6.2 and databricks spark 3.4.0
**URL:** https://discuss.elastic.co/t/connector-for-elastic-search-8-6-2-and-databricks-spark-3-4-0/342746
**Category:** Elasticsearch
**Tags:** es-hadoop
**Created:** [September 11, 2023, 2:40pm UTC](https://discuss.elastic.co/t/connector-for-elastic-search-8-6-2-and-databricks-spark-3-4-0/342746 "2023-09-11T14:40:29Z")
**Posts on this page:** 10
**Page:** 1
### Author:  [@luckymishra](https://discuss.elastic.co/u/luckymishra)
#### Post date: [September 11, 2023, 2:40pm UTC](https://discuss.elastic.co/t/connector-for-elastic-search-8-6-2-and-databricks-spark-3-4-0/342746/1 "2023-09-11T14:40:29Z")
I am trying to connect Elastic Search using spark in data bricks but keep getting this error org.elasticsearch.hadoop.EsHadoopIllegalArgumentException: Cannot detect ES version - typically this happens if the network/Elasticsearch cluster is not accessible or when targeting a WAN/Cloud instance without the proper setting 'es.nodes.wan.only'.
The connector i installed is "org.elasticsearch:elasticsearch-spark-30\_2.12:7.17.13"
Spark version on data bricks is 3.4.0
Elastic Search version is 8.6.2 and scala version on data bricks is 2.12.15
Any suggestion will be appreciated.
Thanks
---
### Author:  [@Keith\_Massey](https://discuss.elastic.co/u/Keith_Massey)
#### Post date: [September 13, 2023, 12:50pm UTC](https://discuss.elastic.co/t/connector-for-elastic-search-8-6-2-and-databricks-spark-3-4-0/342746/2 "2023-09-13T12:50:35Z")
Hi @luckymishra. You'll want your es-spark version to be compatible with elasticsearch 8.x. So you probably want `org.elasticsearch:elasticsearch-spark-30_2.12:8.6.2`.
---
### Author:  [@luckymishra](https://discuss.elastic.co/u/luckymishra)
#### Post date: [September 14, 2023, 9:33pm UTC](https://discuss.elastic.co/t/connector-for-elastic-search-8-6-2-and-databricks-spark-3-4-0/342746/3 "2023-09-14T21:33:50Z")
thank you, i could add the library but still getting this error org.elasticsearch.hadoop.EsHadoopIllegalArgumentException: Cannot detect ES version - typically this happens if the network/Elasticsearch cluster is not accessible or when targeting a WAN/Cloud instance without the proper setting 'es.nodes.wan.only
---
### Author:  [@Keith\_Massey](https://discuss.elastic.co/u/Keith_Massey)
#### Post date: [September 14, 2023, 9:55pm UTC](https://discuss.elastic.co/t/connector-for-elastic-search-8-6-2-and-databricks-spark-3-4-0/342746/4 "2023-09-14T21:55:55Z")
So it's working with `es.nodes.wan.only` set to true, and the list of nodes given? And you're wanting to run with that set to false instead? Are all of your spark nodes able to access all of your elasticsearch nodes?
---
### Author:  [@luckymishra](https://discuss.elastic.co/u/luckymishra)
#### Post date: [September 14, 2023, 10:06pm UTC](https://discuss.elastic.co/t/connector-for-elastic-search-8-6-2-and-databricks-spark-3-4-0/342746/5 "2023-09-14T22:06:56Z")
i want to send you the code but the system is not letting me send the code .
no setting es.nodes.wan.only to true/false doesn't make any impact
just to let you know it's work perfectly if i call the connection like es\_client=connections.create\_connection(cloud\_id= 'xxxxxxx',api\_key= 'xxxxxx') but this can't be used with spark
---
### Author:  [@Keith\_Massey](https://discuss.elastic.co/u/Keith_Massey)
#### Post date: [September 14, 2023, 10:52pm UTC](https://discuss.elastic.co/t/connector-for-elastic-search-8-6-2-and-databricks-spark-3-4-0/342746/6 "2023-09-14T22:52:54Z")
> i want to send you the code but the system is not letting me send the code
You've tried just pasting it in here in a formatted block? Maybe it's too much?
> just to let you know it's work perfectly if i call the connection like es\_client=connections.create\_connection(cloud\_id= 'xxxxxxx',api\_key= 'xxxxxx') but this can't be used with spark
I'm not familiar with `cloud_id`. What kind of authentication are you using? I assume you've seen [ES-Hadoop and Security | Elasticsearch Guide [8.11] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/hadoop.html)?
---
### Author:  [@luckymishra](https://discuss.elastic.co/u/luckymishra)
#### Post date: [September 29, 2023, 1:49pm UTC](https://discuss.elastic.co/t/connector-for-elastic-search-8-6-2-and-databricks-spark-3-4-0/342746/7 "2023-09-29T13:49:30Z")
Hi Keith, apologies for not coming back, just to let you know i am still getting error which means the connector version is not correct. i am using the standard template to connect but the only difference is that elastic is hosted on cloud so i have address starting from https: , can you suggest something ?
org.elasticsearch.hadoop.EsHadoopIllegalArgumentException: Cannot detect ES version - typically this happens if the network/Elasticsearch cluster is not accessible or when targeting a WAN/Cloud instance without the proper setting 'es.nodes.wan.only'
---
### Author:  [@Keith\_Massey](https://discuss.elastic.co/u/Keith_Massey)
#### Post date: [September 29, 2023, 3:42pm UTC](https://discuss.elastic.co/t/connector-for-elastic-search-8-6-2-and-databricks-spark-3-4-0/342746/8 "2023-09-29T15:42:29Z")
Have you set `es.nodes.wan.only` to true? Using es-hadoop/spark through cloud, elasticsearch appears to be a single node to es-hadoop. You might want to take a look at [Writing PySpark dataframe to Elastic Cloud (Cannot detect ES version)](https://discuss.elastic.co/t/writing-pyspark-dataframe-to-elastic-cloud-cannot-detect-es-version/334176) for a discussion about some of the complications that come with that.
---
### Author:  [@luckymishra](https://discuss.elastic.co/u/luckymishra)
#### Post date: [September 29, 2023, 3:53pm UTC](https://discuss.elastic.co/t/connector-for-elastic-search-8-6-2-and-databricks-spark-3-4-0/342746/9 "2023-09-29T15:53:14Z")
yes i set it to true, ok will go through the link
es\_conf = {"es.nodes": "xxxx", "es.port": 9243, "es.nodes.wan.only": True }
es\_conf["es.cloud.id"] = "xxxx"
es\_conf["es.api.key"] = "xxx"
es\_conf["es.net.ssl"] = "true"
df = spark.read.format("org.elasticsearch.spark.sql").options(\*\*es\_conf).load("abc")
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [October 27, 2023, 3:54pm UTC](https://discuss.elastic.co/t/connector-for-elastic-search-8-6-2-and-databricks-spark-3-4-0/342746/10 "2023-10-27T15:54:04Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.