# Pyspark - read data from elasticsearch cluster on EMR

**URL:** <https://discuss.elastic.co/t/pyspark-read-data-from-elasticsearch-cluster-on-emr/158454>\
**Category:** Elasticsearch\
**Tags:** es-hadoop\
**Created:** [November 28, 2018, 4:44am UTC](https://discuss.elastic.co/t/pyspark-read-data-from-elasticsearch-cluster-on-emr/158454 "2018-11-28T04:44:55Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![ruxiz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruxiz/32/38157_2.png) [@ruxiz](https://discuss.elastic.co/u/ruxiz)\
**Post date:** [November 28, 2018, 4:44am UTC](https://discuss.elastic.co/t/pyspark-read-data-from-elasticsearch-cluster-on-emr/158454/1 "2018-11-28T04:44:55Z")

</div>

I am trying to read data from elasticsearch from pyspark. I was using the elasticsearch-hadoop api in Spark. The es cluster sits on aws emr, which requires credential to sign in. My script is as below:

> from pyspark import SparkContext, SparkConf  
> sc.stop()  
> conf = SparkConf().setAppName("ESTest")  
> sc = SparkContext(conf=conf)  
> es\_read\_conf = {  
> "es.host" : "vhost",  
> "es.nodes" : "node",  
> "es.port" : "443",  
> #"es.query": '{ "query": { "match\_all": {} } }',  
> #"es.input.json": "true",  
> "es.net.https.auth.user": "aws\_access\_key",  
> "es.net.https.auth.pass": "aws\_secret\_key",  
> "es.net.ssl": "true",  
> "es.resource" : "index/type",  
> "es.nodes.wan.only": "true"  
> }
> 
> es\_rdd = sc.newAPIHadoopRDD(  
> inputFormatClass="org.elasticsearch.hadoop.mr.EsInputFormat",  
> keyClass="org.apache.hadoop.io.NullWritable",  
> valueClass="org.elasticsearch.hadoop.mr.LinkedMapWritable",  
> conf=es\_read\_conf)

Pyspark keeps throwing error:

**py4j.protocol.Py4JJavaError: An error occurred while calling z:org.apache.spark.api.python.PythonRDD.newAPIHadoopRDD.**

**: org.elasticsearch.hadoop.rest.EsHadoopInvalidRequest: [HEAD] on [index] failed; servernode:443] returned [403|Forbidden:]**

---

<div class="post-metadata">

**Author:** ![ruxiz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruxiz/32/38157_2.png) [@ruxiz](https://discuss.elastic.co/u/ruxiz)\
**Post date:** [November 28, 2018, 4:47am UTC](https://discuss.elastic.co/t/pyspark-read-data-from-elasticsearch-cluster-on-emr/158454/2 "2018-11-28T04:47:02Z")

</div>

I checked everything which all made sense except for the user and pass entries, would aws access key and secret key work here? We don't want to use the console user and password here for security purpose. Please advice! Thanks!

---

<div class="post-metadata">

**Author:** ![ruxiz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruxiz/32/38157_2.png) [@ruxiz](https://discuss.elastic.co/u/ruxiz)\
**Post date:** [November 28, 2018, 5:40pm UTC](https://discuss.elastic.co/t/pyspark-read-data-from-elasticsearch-cluster-on-emr/158454/3 "2018-11-28T17:40:43Z")

</div>

Looks like this client just does not support AWS connector, and I need a custom AWS connect to connect to AWS EMR cluster. DO you have any suggestion what way I should do it?

---

<div class="post-metadata">

**Author:** ![james.baiera](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/james.baiera/32/10209_2.png) [@james.baiera](https://discuss.elastic.co/u/james.baiera)\
**Post date:** [December 12, 2018, 7:48pm UTC](https://discuss.elastic.co/t/pyspark-read-data-from-elasticsearch-cluster-on-emr/158454/4 "2018-12-12T19:48:27Z")

</div>

I am not too familiar with integrating with technologies in AWS since we do not test against those environments at all in the connector. I will say that the `es.net.https.auth.user` and `es.net.https.auth.pass` configurations are packaged together as a `Basic` Http authentication scheme header in Base64 format. If you need alternative headers to be sent, I would specify them by setting them in the configuration: [https://www.elastic.co/guide/en/elasticsearch/hadoop/current/configuration.html#\_setting\_http\_request\_headers](https://www.elastic.co/guide/en/elasticsearch/hadoop/current/configuration.html#_setting_http_request_headers)

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [January 9, 2019, 7:48pm UTC](https://discuss.elastic.co/t/pyspark-read-data-from-elasticsearch-cluster-on-emr/158454/5 "2019-01-09T19:48:34Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.