# Problems connecting to ES from Databricks using spark connector

**URL:** <https://discuss.elastic.co/t/problems-connecting-to-es-from-databricks-using-spark-connector/332411>\
**Category:** Elasticsearch\
**Tags:** es-hadoop\
**Created:** [May 3, 2023, 11:34am UTC](https://discuss.elastic.co/t/problems-connecting-to-es-from-databricks-using-spark-connector/332411 "2023-05-03T11:34:11Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![lhfo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lhfo/32/120573_2.png) [@lhfo](https://discuss.elastic.co/u/lhfo)\
**Post date:** [May 3, 2023, 11:34am UTC](https://discuss.elastic.co/t/problems-connecting-to-es-from-databricks-using-spark-connector/332411/1 "2023-05-03T11:34:11Z")

</div>

Hi all,

I am trying to connect with ES from our Databricks cluster.

I have successfully installed `elasticsearch-spark-30_2.12:8.4.3` on the cluster and confirmed that the elastic version == 8.3.4.

I am able to query data from ES using post commands e.g.

```auto
curl --user 'username:password' -X GET 'https://my-elasitic-url.elastic-cloud.com/index/_search' -H 'Accept: application/json' -H 'Content-type: application/json' -d '{"query": {"query_string": {"query": "my_query"}}}'

```

works.

however, running the following code

```auto
df = (spark.read
      .format( "org.elasticsearch.spark.sql" )
      .option( "es.nodes", https://my-elasitic-url.elastic-cloud.com)
      .option( "es.net.ssl", "false")
      .option( "es.nodes.wan.only", "true" )
      .option( "es.net.http.auth.user", username)
      .option( "es.net.http.auth.pass", password)
      .load(headers["index"])
     )

```

results in this error.

```auto
org.elasticsearch.hadoop.EsHadoopIllegalArgumentException: Cannot detect ES version - typically this happens if the network/Elasticsearch cluster is not accessible or when targeting a WAN/Cloud instance without the proper setting 'es.nodes.wan.only'

```

any suggestions?

---

<div class="post-metadata">

**Author:** ![Keith\_Massey](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/keith_massey/32/83666_2.png) [@Keith\_Massey](https://discuss.elastic.co/u/Keith_Massey)\
**Post date:** [May 3, 2023, 1:54pm UTC](https://discuss.elastic.co/t/problems-connecting-to-es-from-databricks-using-spark-connector/332411/2 "2023-05-03T13:54:53Z")

</div>

Hi @lhfo . What is the full stack trace? There ought to be a "Caused by" that will tell us a little more. Also, is your elasticsearch cluster accessible from the node where your spark driver is running?

---

<div class="post-metadata">

**Author:** ![lhfo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lhfo/32/120573_2.png) [@lhfo](https://discuss.elastic.co/u/lhfo)\
**Post date:** [May 5, 2023, 8:52am UTC](https://discuss.elastic.co/t/problems-connecting-to-es-from-databricks-using-spark-connector/332411/3 "2023-05-05T08:52:52Z")

</div>

I solved this issue by setting

```auto
 .option("es.port","443")
 .option( "es.net.ssl", "true")

```

@Keith_Massey , thanks for the reply!

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [June 5, 2023, 6:06am UTC](https://discuss.elastic.co/t/problems-connecting-to-es-from-databricks-using-spark-connector/332411/5 "2023-06-05T06:06:48Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.