# ElasticSearch Spark Hadoop Connector

**URL:** <https://discuss.elastic.co/t/elasticsearch-spark-hadoop-connector/37156>\
**Category:** Elasticsearch\
**Tags:** es-hadoop\
**Created:** [December 14, 2015, 7:17pm UTC](https://discuss.elastic.co/t/elasticsearch-spark-hadoop-connector/37156 "2015-12-14T19:17:13Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![Pinhus\_Dash](https://avatars.discourse-cdn.com/v4/letter/p/b9bd4f/32.png) [@Pinhus\_Dash](https://discuss.elastic.co/u/Pinhus_Dash)\
**Post date:** [December 14, 2015, 7:17pm UTC](https://discuss.elastic.co/t/elasticsearch-spark-hadoop-connector/37156/1 "2015-12-14T19:17:13Z")

</div>

Continuing the discussion from [EsHadoopInvalidRequest: [POST]](https://discuss.elastic.co/t/eshadoopinvalidrequest-post/31635/9):

> [@EsHadoopInvalidRequest: \[POST\]](https://discuss.elastic.co/t/eshadoopinvalidrequest-post/31635/9):
>
> I am having issues with the ES Spark Connector.
> 
> When I connect to 1 remote machine ES cluster, I get the following error  
> I can read an index, and I get the schema from the dataframe created with this command:
> 
> val options = Map("pushdown" -\> "true", "es.nodes" -\> "machine1", "es.port" -\> "9200")  
> sqlContext.read.format("org.elasticsearch.spark.sql").options(options).load("index/type")
> 
> org.elasticsearch.hadoop.rest.EsHadoopInvalidRequest: ElasticsearchIllegalArgumentException[No data node with id[5\_JV4ZDkRweIZcP79cNNOA] found]
> 
> When I connect to a different remote machine ES cluster, I can read the schema from ES.  
> Example:  
> val optionsDev = Map("pushdown" -\> "true", "es.nodes" -\> "machineDev", "es.port" -\> "9200")  
> val spark14DF = sqlContext.read.format("org.elasticsearch.spark.sql").options(optionsDev).load("blog/post")
> 
> spark14DF: org.apache.spark.sql.DataFrame = [body: string, postDate: timestamp, title: string, user: string]
> 
> spark14DF.printSchema  
> root  
> |-- body: string (nullable = true)  
> |-- postDate: timestamp (nullable = true)  
> |-- title: string (nullable = true)  
> |-- user: string (nullable = true)
> 
> But when I ask for spark14DF.first()  
> I get the following error:  
> org.elasticsearch.hadoop.rest.EsHadoopInvalidRequest: IndexMissingException[[blog] missing]  
> at org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:352)  
> .....

Theses same commands work fine when I am using the localhost elasticsearch cluster which consists of three nodes.  
The previous two example only use one node. But they each give different errors. So there must be something else I don't understand.

Thank you

---

<div class="post-metadata">

**Author:** ![costin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/costin/32/44950_2.png) [@costin](https://discuss.elastic.co/u/costin)\
**Post date:** [December 18, 2015, 12:39pm UTC](https://discuss.elastic.co/t/elasticsearch-spark-hadoop-connector/37156/2 "2015-12-18T12:39:18Z")

</div>

Likely your localhost cluster is configured differently then the remote clusters.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 1:26pm UTC](https://discuss.elastic.co/t/elasticsearch-spark-hadoop-connector/37156/3 "2017-07-06T13:26:53Z")

</div>