Pyspark - read data from elasticsearch cluster on EMR

Elastic Stack Elasticsearch
1

I am trying to read data from elasticsearch from pyspark. I was using the elasticsearch-hadoop api in Spark. The es cluster sits on aws emr, which requires credential to sign in. My script is as below:

from pyspark import SparkContext, SparkConf
sc.stop()
conf = SparkConf().setAppName("ESTest")
sc = SparkContext(conf=conf)
es_read_conf = {
"es.host" : "vhost",
"es.nodes" : "node",
"es.port" : "443",
#"es.query": '{ "query": { "match_all": {} } }',
#"es.input.json": "true",
"es.net.https.auth.user": "aws_access_key",
"es.net.https.auth.pass": "aws_secret_key",
"es.net.ssl": "true",
"es.resource" : "index/type",
"es.nodes.wan.only": "true"
}

es_rdd = sc.newAPIHadoopRDD(
inputFormatClass="org.elasticsearch.hadoop.mr.EsInputFormat",
keyClass="org.apache.hadoop.io.NullWritable",
valueClass="org.elasticsearch.hadoop.mr.LinkedMapWritable",
conf=es_read_conf)

Pyspark keeps throwing error:

py4j.protocol.Py4JJavaError: An error occurred while calling z:org.apache.spark.api.python.PythonRDD.newAPIHadoopRDD.

: org.elasticsearch.hadoop.rest.EsHadoopInvalidRequest: [HEAD] on [index] failed; servernode:443] returned [403|Forbidden:]

2

I checked everything which all made sense except for the user and pass entries, would aws access key and secret key work here? We don't want to use the console user and password here for security purpose. Please advice! Thanks!

3

Looks like this client just does not support AWS connector, and I need a custom AWS connect to connect to AWS EMR cluster. DO you have any suggestion what way I should do it?

4

I am not too familiar with integrating with technologies in AWS since we do not test against those environments at all in the connector. I will say that the es.net.https.auth.user and es.net.https.auth.pass configurations are packaged together as a Basic Http authentication scheme header in Base64 format. If you need alternative headers to be sent, I would specify them by setting them in the configuration: https://www.elastic.co/guide/en/elasticsearch/hadoop/current/configuration.html#_setting_http_request_headers