# Error writing to Elastic search from Databricks **URL:** \ **Category:** Elasticsearch\ **Tags:** es-hadoop\ **Created:** [February 7, 2024, 3:07am UTC](https://discuss.elastic.co/t/error-writing-to-elastic-search-from-databricks/352696 "2024-02-07T03:07:28Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![kichcha](https://avatars.discourse-cdn.com/v4/letter/k/a88e4f/32.png) [@kichcha](https://discuss.elastic.co/u/kichcha)\ **Post date:** [February 7, 2024, 3:07am UTC](https://discuss.elastic.co/t/error-writing-to-elastic-search-from-databricks/352696/1 "2024-02-07T03:07:28Z")
Hello, I am an Elasticsearch newbie trying to connect to Elasticsearch on GCP from databricks on AWS. I tried following instructions provided by databricks (unable to post link here). However, I am now running into the following error: org.elasticsearch.hadoop.EsHadoopIllegalArgumentException: Cannot detect ES version - typically this happens if the network/Elasticsearch cluster is not accessible or when targeting a WAN/Cloud instance without the proper setting 'es.nodes.wan.only' I am able to ping the elastic instance from databricks. I am running on cluster databricks run time LTS 13.3 which uses Scala 2.12 and Spark 3.4.1. I have installed elasticsearch-spark-30\_2.12:8.11.4 library from maven repo. My code is: ```auto ( people.write .format("org.elasticsearch.spark.sql") .option("es.nodes", "35.193.143.25:9243") .option("es.index.auto.create", "true") .option("es.resource", "test") .option("es.nodes.wan.only", "true") .mode("overwrite") .save() ) ``` Appreciate any help on resolving this issue. Thank you! ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [February 7, 2024, 3:18am UTC](https://discuss.elastic.co/t/error-writing-to-elastic-search-from-databricks/352696/2 "2024-02-07T03:18:03Z")
Which version of Elasticsearch are you using? How is this configured? ---
**Author:** ![kichcha](https://avatars.discourse-cdn.com/v4/letter/k/a88e4f/32.png) [@kichcha](https://discuss.elastic.co/u/kichcha)\ **Post date:** [February 7, 2024, 3:59am UTC](https://discuss.elastic.co/t/error-writing-to-elastic-search-from-databricks/352696/3 "2024-02-07T03:59:59Z")
I am using Elasticsearch 8.12.1 provisioned in GCP through elastic cloud. This is a very basic trial version of the service. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [February 7, 2024, 6:56am UTC](https://discuss.elastic.co/t/error-writing-to-elastic-search-from-databricks/352696/4 "2024-02-07T06:56:55Z")
All Elastic Cloud clusters have security enabled and it does not seem like you are providing any security settings. I have not used the ES Hadoop connector but have looked at a few related issues and they all have `.option("es.net.ssl", "true")`, `.option("es.net.http.auth.user", "elastic")` and `.option("es.net.http.auth.pass", "password")` set. Can you try adding this to see if it makes any difference? ---
**Author:** ![kichcha](https://avatars.discourse-cdn.com/v4/letter/k/a88e4f/32.png) [@kichcha](https://discuss.elastic.co/u/kichcha)\ **Post date:** [February 7, 2024, 6:30pm UTC](https://discuss.elastic.co/t/error-writing-to-elastic-search-from-databricks/352696/5 "2024-02-07T18:30:51Z")
Thank you for pointing that out. I was thinking about security as well, but, I did not see any options pertaining to this while creating the instance. So, is the user and password "elastic" and "password" or do I need to set this user up in my elastic instance? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [February 7, 2024, 6:34pm UTC](https://discuss.elastic.co/t/error-writing-to-elastic-search-from-databricks/352696/6 "2024-02-07T18:34:06Z")
The username and password will depend on your cluster. I would suggest creating a new user with the correct privilages and use this. The values I provided are only examples. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [March 6, 2024, 6:34pm UTC](https://discuss.elastic.co/t/error-writing-to-elastic-search-from-databricks/352696/7 "2024-03-06T18:34:41Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.