# \#es-hadoop **URL:** https://discuss.elastic.co/tag/es-hadoop/50.md [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) --- ## [What version of elasticsearch-hadoop should I be using with spark 4.0.1](https://discuss.elastic.co/t/what-version-of-elasticsearch-hadoop-should-i-be-using-with-spark-4-0-1/382679)
**Author:** [@Thomas\_Butterfield](https://discuss.elastic.co/u/Thomas_Butterfield)\ **Replies:** 1\ **Last updated:** [October 14, 2025, 1:13pm UTC](https://discuss.elastic.co/t/what-version-of-elasticsearch-hadoop-should-i-be-using-with-spark-4-0-1/382679 "2025-10-14T13:13:53Z")
Using Scala version 2.13.16 (OpenJDK 64-Bit Server VM, Java 17.0.16) I'm getting this error when writing to an ES 8.19.3 cluster: java.lang.NoSuchMethodError: 'org.apache.spark.sql.SQLContext org.apache.spark.sql.Datase… --- ## [Why doesn’t dense\_vector field show up in Spark schema when using Elasticsearch-Hadoop?](https://discuss.elastic.co/t/why-doesn-t-dense-vector-field-show-up-in-spark-schema-when-using-elasticsearch-hadoop/381715)
**Author:** [@dany\_fard](https://discuss.elastic.co/u/dany_fard)\ **Replies:** 2\ **Last updated:** [September 9, 2025, 1:03pm UTC](https://discuss.elastic.co/t/why-doesn-t-dense-vector-field-show-up-in-spark-schema-when-using-elasticsearch-hadoop/381715 "2025-09-09T13:03:46Z")
Hi everyone, I created an Elasticsearch index with a dense\_vector field, along with some text fields. The mapping looks like this (simplified): {"mappings": {"properties": {"embedding": {"type": "dense\_vector","dims": … --- ## [Elasticsearch-Pyspark Problem](https://discuss.elastic.co/t/elasticsearch-pyspark-problem/372495)
**Author:** [@tmslara.a](https://discuss.elastic.co/u/tmslara.a)\ **Replies:** 2\ **Last updated:** [January 20, 2025, 6:48pm UTC](https://discuss.elastic.co/t/elasticsearch-pyspark-problem/372495 "2025-01-20T18:48:52Z")
Hello, I'm trying to connect PySpark with Elasticsearch so that I can manipulate Elasticsearch docs using PySpark. However, I'm currently facing issues when trying to retrieve documents. I've tried different approaches,… --- ## [PySpark writing to ES: "Cannot detect ES version"](https://discuss.elastic.co/t/pyspark-writing-to-es-cannot-detect-es-version/363662)
**Author:** [@rzotti-sd](https://discuss.elastic.co/u/rzotti-sd)\ **Replies:** 9\ **Last updated:** [July 24, 2024, 11:55pm UTC](https://discuss.elastic.co/t/pyspark-writing-to-es-cannot-detect-es-version/363662 "2024-07-24T23:55:44Z")
I'm using the following PySpark code to write a DataFrame to an Elasticsearch cluster hosted in Elastic Cloud. df.writeStream.outputMode("append") .format("org.elasticsearch.spark.sql") .option("checkpoi… --- ## [How to assign weight attribute to a completion suggester field using Elasticsearch Hadoop connector (Spark/Databricks)](https://discuss.elastic.co/t/how-to-assign-weight-attribute-to-a-completion-suggester-field-using-elasticsearch-hadoop-connector-spark-databricks/358815)
**Author:** [@landlord\_matt](https://discuss.elastic.co/u/landlord_matt)\ **Replies:** 0\ **Last updated:** [May 6, 2024, 11:17am UTC](https://discuss.elastic.co/t/how-to-assign-weight-attribute-to-a-completion-suggester-field-using-elasticsearch-hadoop-connector-spark-databricks/358815 "2024-05-06T11:17:28Z")
Hi! We're successfully using the org.elasticsearch.spark.sql driver to upload our Databricks Spark tables as Elasticsearch indices. We now want to assign the weight attribute to one of our suggester fields in order to g… --- ## [Save data from error handler to other index](https://discuss.elastic.co/t/save-data-from-error-handler-to-other-index/357094)
**Author:** [@Observador](https://discuss.elastic.co/u/Observador)\ **Replies:** 0\ **Last updated:** [April 9, 2024, 4:56pm UTC](https://discuss.elastic.co/t/save-data-from-error-handler-to-other-index/357094 "2024-04-09T16:56:20Z")
Hi, I’d like to make an update and save thouse which doesn’t exist in other index. I’ve done an Error Handler but I don’t know how make the call there to write each not existing record data in other index. --- ## [Calling union on two dataframes from spark and elastic search stuck](https://discuss.elastic.co/t/calling-union-on-two-dataframes-from-spark-and-elastic-search-stuck/353599)
**Author:** [@xsa\_xsa](https://discuss.elastic.co/u/xsa_xsa)\ **Replies:** 7\ **Last updated:** [March 1, 2024, 6:50am UTC](https://discuss.elastic.co/t/calling-union-on-two-dataframes-from-spark-and-elastic-search-stuck/353599 "2024-03-01T06:50:39Z")
i am using pyspark to query elstic search i have the folowing code d1 = data\_frame1.collect() # return 1 row d2 = data\_frame2.collect() # return no rows but when i call d3 data\_frame1.union(data\_frame2).collect() t… --- ## [Error writing to Elastic search from Databricks](https://discuss.elastic.co/t/error-writing-to-elastic-search-from-databricks/352696)
**Author:** [@kichcha](https://discuss.elastic.co/u/kichcha)\ **Replies:** 5\ **Last updated:** [February 7, 2024, 6:34pm UTC](https://discuss.elastic.co/t/error-writing-to-elastic-search-from-databricks/352696 "2024-02-07T18:34:06Z")
Hello, I am an Elasticsearch newbie trying to connect to Elasticsearch on GCP from databricks on AWS. I tried following instructions provided by databricks (unable to post link here). However, I am now running into th… --- ## [Error while running pyspark connecting to elastic search](https://discuss.elastic.co/t/error-while-running-pyspark-connecting-to-elastic-search/351195)
**Author:** [@kashi\_mn](https://discuss.elastic.co/u/kashi_mn)\ **Replies:** 2\ **Last updated:** [January 24, 2024, 5:03am UTC](https://discuss.elastic.co/t/error-while-running-pyspark-connecting-to-elastic-search/351195 "2024-01-24T05:03:36Z")
I am new to Pyspark and currently running pyspark and trying to connect to elasticsearch running on localhost. Below are the details : Elastic Details : localhost Port : 9200 No https. Code : from pyspark.sql impor… --- ## [org.elasticsearch.hadoop.rest.EsHadoopRemoteException: illegal\_argument\_exception: value for key \[X-Opaque-Id\] already present](https://discuss.elastic.co/t/org-elasticsearch-hadoop-rest-eshadoopremoteexception-illegal-argument-exception-value-for-key-x-opaque-id-already-present/347380)
**Author:** [@Akhil\_parmar](https://discuss.elastic.co/u/Akhil_parmar)\ **Replies:** 2\ **Last updated:** [November 23, 2023, 3:18am UTC](https://discuss.elastic.co/t/org-elasticsearch-hadoop-rest-eshadoopremoteexception-illegal-argument-exception-value-for-key-x-opaque-id-already-present/347380 "2023-11-23T03:18:13Z")
I am working on ETL job where target to load data to elasticsearch, I am facing an error when trying to index document org.elasticsearch.hadoop.rest.EsHadoopRemoteException: illegal\_argument\_exception: value for key \[X-… --- ## [Add Es Spark Accumulators](https://discuss.elastic.co/t/add-es-spark-accumulators/347127)
**Author:** [@glegoux](https://discuss.elastic.co/u/glegoux)\ **Replies:** 2\ **Last updated:** [November 21, 2023, 9:30pm UTC](https://discuss.elastic.co/t/add-es-spark-accumulators/347127 "2023-11-21T21:30:41Z")
Hello :wave:, I did a pull request. These metrics will be very useful to monitor a Spark application using the extension Elasticsearch for Hadoop. What do you think about it? --- ## [Elasticsearch spark runtime dependencies](https://discuss.elastic.co/t/elasticsearch-spark-runtime-dependencies/344341)
**Author:** [@krezno](https://discuss.elastic.co/u/krezno)\ **Replies:** 1\ **Last updated:** [October 4, 2023, 1:00pm UTC](https://discuss.elastic.co/t/elasticsearch-spark-runtime-dependencies/344341 "2023-10-04T13:00:22Z")
Hello, I have sucessfuly managed to use elasticsearch-spark with both pyspark and scala spark by simply adding the jar to the classpath. I have noticed that the jar has some runtime dependencies such as protobuf-java an… --- ## [Connector for Elastic Search 8.6.2 and databricks spark 3.4.0](https://discuss.elastic.co/t/connector-for-elastic-search-8-6-2-and-databricks-spark-3-4-0/342746)
**Author:** [@luckymishra](https://discuss.elastic.co/u/luckymishra)\ **Replies:** 8\ **Last updated:** [September 29, 2023, 3:53pm UTC](https://discuss.elastic.co/t/connector-for-elastic-search-8-6-2-and-databricks-spark-3-4-0/342746 "2023-09-29T15:53:14Z")
I am trying to connect Elastic Search using spark in data bricks but keep getting this error org.elasticsearch.hadoop.EsHadoopIllegalArgumentException: Cannot detect ES version - typically this happens if the network/Ela… --- ## [Best way to write from Apache Spark to ECK](https://discuss.elastic.co/t/best-way-to-write-from-apache-spark-to-eck/342480)
**Author:** [@krezno](https://discuss.elastic.co/u/krezno)\ **Replies:** 2\ **Last updated:** [September 9, 2023, 9:26pm UTC](https://discuss.elastic.co/t/best-way-to-write-from-apache-spark-to-eck/342480 "2023-09-09T21:26:04Z")
Hello I have a lot of batch processes that write large batches of data to elastic in scheduled intervals. Currently we are writing to elastic using the es-hadoop library. From what I understand when writing to an ECK in… --- ## [EsHadoopIllegalArgumentException: Cannot detect ES version](https://discuss.elastic.co/t/eshadoopillegalargumentexception-cannot-detect-es-version/342593)
**Author:** [@Piyush\_Jain](https://discuss.elastic.co/u/Piyush_Jain)\ **Replies:** 1\ **Last updated:** [September 8, 2023, 1:33pm UTC](https://discuss.elastic.co/t/eshadoopillegalargumentexception-cannot-detect-es-version/342593 "2023-09-08T13:33:00Z")
I'm getting error "EsHadoopIllegalArgumentException: Cannot detect ES version-typically this happens if the network/Elasticsearch cluster is not accessible or when targeting a WAN/Cloud instance without the proper setti… --- ## [Stability issues when uploading Databricks tables to Elasticsearch indices (circuit\_breaking\_exception)](https://discuss.elastic.co/t/stability-issues-when-uploading-databricks-tables-to-elasticsearch-indices-circuit-breaking-exception/340637)
**Author:** [@landlord\_matt](https://discuss.elastic.co/u/landlord_matt)\ **Replies:** 6\ **Last updated:** [August 14, 2023, 3:00pm UTC](https://discuss.elastic.co/t/stability-issues-when-uploading-databricks-tables-to-elasticsearch-indices-circuit-breaking-exception/340637 "2023-08-14T15:00:11Z")
Hi! We recently got switched from a large (8 node, 21 GB JVM heap memory) cluster to a smaller (3 node, 9 GB JVM heap memory) cluster and now we getting errors while trying to reupload our Elasticsearch indices. It occa… --- ## [EsHadoopRemoteException: illegal\_argument\_exception: only write ops with an op\_type of create are allowed in data streams?](https://discuss.elastic.co/t/eshadoopremoteexception-illegal-argument-exception-only-write-ops-with-an-op-type-of-create-are-allowed-in-data-streams/340267)
**Author:** [@Zephery\_Wen](https://discuss.elastic.co/u/Zephery_Wen)\ **Replies:** 4\ **Last updated:** [August 9, 2023, 1:07pm UTC](https://discuss.elastic.co/t/eshadoopremoteexception-illegal-argument-exception-only-write-ops-with-an-op-type-of-create-are-allowed-in-data-streams/340267 "2023-08-09T13:07:05Z")
I want to use Spark to save data to a data stream. JavaEsSparkSQL.saveToEs(result, dataStream, map); It seems like 'saveToEs' only allows index. EsHadoopRemoteException: illegal\_argument\_exception: only write ops with… --- ## [EsHadoopIllegalArgumentException: Detected type name in resource On elastic 8.8.2](https://discuss.elastic.co/t/eshadoopillegalargumentexception-detected-type-name-in-resource-on-elastic-8-8-2/339660)
**Author:** [@Joachim\_Rodrigues](https://discuss.elastic.co/u/Joachim_Rodrigues)\ **Replies:** 1\ **Last updated:** [July 31, 2023, 9:27pm UTC](https://discuss.elastic.co/t/eshadoopillegalargumentexception-detected-type-name-in-resource-on-elastic-8-8-2/339660 "2023-07-31T21:27:19Z")
Hi I upgraded my elastic server fom 7.9.3 to 8.8.2 With my previous configuration : implementation("org.elasticsearch:elasticsearch-spark-20\_2.11:7.9.3") { exclude("org.apache.spark") } This code was … --- ## [Bulk insert with Spark causes org.elasticsearch.hadoop.rest.EsHadoopNoNodesLeftException: Connection error (check network and/or proxy settings)- all nodes failed](https://discuss.elastic.co/t/bulk-insert-with-spark-causes-org-elasticsearch-hadoop-rest-eshadoopnonodesleftexception-connection-error-check-network-and-or-proxy-settings-all-nodes-failed/337631)
**Author:** [@Thijsvdp](https://discuss.elastic.co/u/Thijsvdp)\ **Replies:** 6\ **Last updated:** [July 6, 2023, 3:17pm UTC](https://discuss.elastic.co/t/bulk-insert-with-spark-causes-org-elasticsearch-hadoop-rest-eshadoopnonodesleftexception-connection-error-check-network-and-or-proxy-settings-all-nodes-failed/337631 "2023-07-06T15:17:06Z")
Hi all, I am having trouble with writing to a 5-node Elasticsearch cluster with Spark. Some basic details about the cluster: 5 nodes 6TB of disk 5x28 GB of RAM (half is Heap) 5x6 CPU Allocated 140 shards for the relev… --- ## [Problems connecting to ES Cross cluster search cluster indexes from Databricks using spark connector](https://discuss.elastic.co/t/problems-connecting-to-es-cross-cluster-search-cluster-indexes-from-databricks-using-spark-connector/335030)
**Author:** [@srinivas\_a1](https://discuss.elastic.co/u/srinivas_a1)\ **Replies:** 3\ **Last updated:** [June 29, 2023, 4:56pm UTC](https://discuss.elastic.co/t/problems-connecting-to-es-cross-cluster-search-cluster-indexes-from-databricks-using-spark-connector/335030 "2023-06-29T16:56:32Z")
Hi All, I am trying to connect with ES from our Databricks cluster using elasticsearch\_spark\_30\_2\_12\_7\_16\_3.jar. I'm not able to read the data from cross cluster indexes which are starting with "\*:xxxxxxx", However able… --- ## [Elasticsearch and Hive integration failure with es-hadoop-connector 8.8.1](https://discuss.elastic.co/t/elasticsearch-and-hive-integration-failure-with-es-hadoop-connector-8-8-1/336423)
**Author:** [@Bob\_Dorous](https://discuss.elastic.co/u/Bob_Dorous)\ **Replies:** 2\ **Last updated:** [June 21, 2023, 10:49am UTC](https://discuss.elastic.co/t/elasticsearch-and-hive-integration-failure-with-es-hadoop-connector-8-8-1/336423 "2023-06-21T10:49:43Z")
Hi there, I am trying to set up the newest release Elasticsearch (8.8.1) as a single-node service on an Ubuntu Azure VM and write and read to it with Hive 3.10+ on Hadoop HDInsight cluster (hortonworks based). For the … --- ## [Multiple ES-Hadoop versions detected in the classpath](https://discuss.elastic.co/t/multiple-es-hadoop-versions-detected-in-the-classpath/335852)
**Author:** [@Suna.Y](https://discuss.elastic.co/u/Suna.Y)\ **Replies:** 3\ **Last updated:** [June 14, 2023, 3:29am UTC](https://discuss.elastic.co/t/multiple-es-hadoop-versions-detected-in-the-classpath/335852 "2023-06-14T03:29:44Z")
ERROR: Multiple ES-Hadoop versions detected - Elastic Stack / Elasticsearch - Discuss the Elastic Stack I met the same question as above. If keep those versions, it occurs Multiple ES-Hadoop versions detected as follow… --- ## [Is there any jar to support elasticsearch integrated to tez on hadoop?](https://discuss.elastic.co/t/is-there-any-jar-to-support-elasticsearch-integrated-to-tez-on-hadoop/335857)
**Author:** [@Suna.Y](https://discuss.elastic.co/u/Suna.Y)\ **Replies:** 1\ **Last updated:** [June 13, 2023, 1:24pm UTC](https://discuss.elastic.co/t/is-there-any-jar-to-support-elasticsearch-integrated-to-tez-on-hadoop/335857 "2023-06-13T13:24:05Z")
Is there any jar to support elasticsearch integrated to tez on hadoop? I didn't see any jar like elasticsearch-hadoop-tez.jar --- ## [I want to use spark to read data from es, but I don't know what es.net.ssl.keystore.pass is](https://discuss.elastic.co/t/i-want-to-use-spark-to-read-data-from-es-but-i-dont-know-what-es-net-ssl-keystore-pass-is/335210)
**Author:** [@gaorui](https://discuss.elastic.co/u/gaorui)\ **Replies:** 3\ **Last updated:** [June 6, 2023, 7:58am UTC](https://discuss.elastic.co/t/i-want-to-use-spark-to-read-data-from-es-but-i-dont-know-what-es-net-ssl-keystore-pass-is/335210 "2023-06-06T07:58:51Z")
When I built the es cluster, I used bin/elasticsearch-certutil to generate the CA certificate and p12 certificate, but I did not enter the password, but chose to press Enter directly. When I want to use spark to connect… --- ## [Elastic - Spark connector failing to read data](https://discuss.elastic.co/t/elastic-spark-connector-failing-to-read-data/334231)
**Author:** [@ljSolaiman](https://discuss.elastic.co/u/ljSolaiman)\ **Replies:** 7\ **Last updated:** [June 1, 2023, 1:57pm UTC](https://discuss.elastic.co/t/elastic-spark-connector-failing-to-read-data/334231 "2023-06-01T13:57:46Z")
Hi all, I am trying to read data from Elasticsearch to Databricks (Spark) but I'm getting the following error: org.elasticsearch.hadoop.EsHadoopIllegalArgumentException: Cannot detect ES version - typically this happen… --- ## [Writing PySpark dataframe to Elastic Cloud (Cannot detect ES version)](https://discuss.elastic.co/t/writing-pyspark-dataframe-to-elastic-cloud-cannot-detect-es-version/334176)
**Author:** [@Dmytro\_Ostapchuk](https://discuss.elastic.co/u/Dmytro_Ostapchuk)\ **Replies:** 6\ **Last updated:** [May 24, 2023, 2:06pm UTC](https://discuss.elastic.co/t/writing-pyspark-dataframe-to-elastic-cloud-cannot-detect-es-version/334176 "2023-05-24T14:06:12Z")
Hi there! My use case Run PySpark job on EMR Serverless that reads data from S3 and writes it into Elastic cloud. Errors Cannot detect ES version - typically this happens if the network/Elasticsearch cluster is not a… --- ## [ES hadoop spark connector having a issue with nested json](https://discuss.elastic.co/t/es-hadoop-spark-connector-having-a-issue-with-nested-json/332551)
**Author:** [@Kuldeep\_Pal](https://discuss.elastic.co/u/Kuldeep_Pal)\ **Replies:** 3\ **Last updated:** [May 4, 2023, 3:52pm UTC](https://discuss.elastic.co/t/es-hadoop-spark-connector-having-a-issue-with-nested-json/332551 "2023-05-04T15:52:26Z")
org.elasticsearch.hadoop.rest.EsHadoopParsingException: org.elasticsearch.hadoop.EsHadoopIllegalStateException: Position for 'contacts.phone' not found in row; typically this is caused by a mapping inconsistency Not abl… --- ## [Problems connecting to ES from Databricks using spark connector](https://discuss.elastic.co/t/problems-connecting-to-es-from-databricks-using-spark-connector/332411)
**Author:** [@lhfo](https://discuss.elastic.co/u/lhfo)\ **Replies:** 2\ **Last updated:** [May 5, 2023, 8:52am UTC](https://discuss.elastic.co/t/problems-connecting-to-es-from-databricks-using-spark-connector/332411 "2023-05-05T08:52:52Z")
Hi all, I am trying to connect with ES from our Databricks cluster. I have successfully installed elasticsearch-spark-30\_2.12:8.4.3 on the cluster and confirmed that the elastic version == 8.3.4. I am able to query d… --- ## [Read from ES index fails without write permission with HEAD \[405|Method Not Allowed:\]](https://discuss.elastic.co/t/read-from-es-index-fails-without-write-permission-with-head-405-method-not-allowed/328176)
**Author:** [@petersedivec](https://discuss.elastic.co/u/petersedivec)\ **Replies:** 10\ **Last updated:** [April 3, 2023, 3:03pm UTC](https://discuss.elastic.co/t/read-from-es-index-fails-without-write-permission-with-head-405-method-not-allowed/328176 "2023-04-03T15:03:29Z")
We're trying to connect to 2 different elastic instances and read data from databricks on indicies where a user has read permissions. For both instances we're seeing the following error EsHadoopInvalidRequest: \[HEAD\] o… --- ## [Elastic-hadoop access failing due to SSL check](https://discuss.elastic.co/t/elastic-hadoop-access-failing-due-to-ssl-check/328857)
**Author:** [@NN1](https://discuss.elastic.co/u/NN1)\ **Replies:** 1\ **Last updated:** [March 29, 2023, 9:27pm UTC](https://discuss.elastic.co/t/elastic-hadoop-access-failing-due-to-ssl-check/328857 "2023-03-29T21:27:18Z")
I am trying to connect to a https ES node through my external table created in hdfs cluster. I connect to this hdfs cluster from my linux server using beeline and created the external table setting the property 'es.net.s… [Next page](https://discuss.elastic.co/tag/es-hadoop/50.md?match_all_tags=true&page=1&tags%5B%5D=es-hadoop)