Unable to connect to AWS hosted elasticsearch from spark hadoop with 403 forbidden error

[Sathish4k (Sathish Subramaniam)]

Im trying to connect to a AWS hosted elastic search and I am able to connect to the ES host, but getting a 403 forbidden error.

This is how I connect to ES from spark shell

val conf = new SparkConf()
  conf.set("spark.es.nodes","https://URL.amazonaws.com")
  conf.set("spark.es.port","PORT")
  conf.set("spark.es.nodes.discovery","false")
  conf.set("spark.es.nodes.client.only","false")
  conf.set("spark.es.nodes.wan.only","true")
  conf.set("spark.es.net.http.auth.user","AWS SECRET ID")
  conf.set("spark.es.net.http.auth.pass","AWS SECRET PASS")
  conf.set("spark.es.resouce.auto.create","XXX")
  
  
  val sc = new SparkContext(conf)
  val sqlContext = new org.apache.spark.sql.SQLContext(sc);    
 
val numbers = Map("one" -> 1, "two" -> 2, "three" -> 3)
val airports = Map("arrival" -> "Otopeni", "SFO" -> "San Fran")

sc.makeRDD(Seq(numbers, airports)).saveToEs("tracking_events/events")

I am getting the below error

18/02/15 08:54:29 DEBUG content: << "{"message":"Authorization header requires 'Credential' parameter. Authorization header requires 'Signature' parameter. Authorization header requires 'SignedHeaders' parameter. Authorization header requires existence of either a 'X-Amz-Date' or a 'Date' header. Authorization=Basic <<<<<<<>>>>>>>>>=="}"
18/02/15 08:54:29 TRACE HeaderElement: enter HeaderElement.parseElements(char[])
18/02/15 08:54:29 TRACE HeaderElement: enter HeaderElement.parseElements(String)
18/02/15 08:54:29 TRACE HeaderElement: enter HeaderElement.parseElements(String)
18/02/15 08:54:29 TRACE HeaderElement: enter HeaderElement.parseElements(char[])
18/02/15 08:54:29 DEBUG HttpMethodBase: Should NOT close connection in response to directive: keep-alive
18/02/15 08:54:29 TRACE HttpConnection: enter HttpConnection.isResponseAvailable()
18/02/15 08:54:29 TRACE HeaderElement: enter HeaderElement.parseElements(char[])
18/02/15 08:54:29 TRACE HttpConnection: enter HttpConnection.releaseConnection()
18/02/15 08:54:29 DEBUG HttpConnection: Releasing connection back to connection manager.
18/02/15 08:54:29 TRACE HttpMethodBase: enter getContentCharSet( Header contentheader )
18/02/15 08:54:29 TRACE HeaderElement: enter HeaderElement.parseElements(String)
18/02/15 08:54:29 TRACE HeaderElement: enter HeaderElement.parseElements(char[])
18/02/15 08:54:29 TRACE HeaderElement: enter HeaderElement.getParameterByName(String)
18/02/15 08:54:29 TRACE HeaderElement: enter HeaderElement.getParameterByName(String)
18/02/15 08:54:29 TRACE HeaderElement: enter HeaderElement.getParameterByName(String)
18/02/15 08:54:29 TRACE HeaderElement: enter HeaderElement.getParameterByName(String)
18/02/15 08:54:29 DEBUG HttpMethodBase: Default charset used: UTF-8
18/02/15 08:54:29 DEBUG HttpMethodBase: Default charset used: UTF-8
18/02/15 08:54:29 DEBUG HttpMethodBase: Default charset used: UTF-8
18/02/15 08:54:29 DEBUG HttpMethodBase: Default charset used: UTF-8
18/02/15 08:54:29 TRACE CommonsHttpTransport: Rx @[x.x.x.x] [403-Forbidden] [{"message":"Authorization header requires 'Credential' parameter. Authorization header requires 'Signature' parameter. Authorization header requires 'SignedHeaders' parameter. Authorization header requires existence of either a 'X-Amz-Date' or a 'Date' header. Authorization=Basic <<<<<<<>>>>>>>>>=="}]
18/02/15 08:54:29 TRACE CommonsHttpTransport: Rx @[x.x.x.x] [403-Forbidden] [{"message":"Authorization header requires 'Credential' parameter. Authorization header requires 'Signature' parameter. Authorization header requires 'SignedHeaders' parameter. Authorization header requires existence of either a 'X-Amz-Date' or a 'Date' header. Authorization=Basic <<<<<<<>>>>>>>>>=="}]
18/02/15 08:54:29 TRACE CommonsHttpTransport: Rx @[x.x.x.x] [403-Forbidden] [{"message":"Authorization header requires 'Credential' parameter. Authorization header requires 'Signature' parameter. Authorization header requires 'SignedHeaders' parameter. Authorization header requires existence of either a 'X-Amz-Date' or a 'Date' header. Authorization=Basic <<<<<<<>>>>>>>>>=="}]
18/02/15 08:54:29 TRACE CommonsHttpTransport: Rx @[x.x.x.x] [403-Forbidden] [{"message":"Authorization header requires 'Credential' parameter. Authorization header requires 'Signature' parameter. Authorization header requires 'SignedHeaders' parameter. Authorization header requires existence of either a 'X-Amz-Date' or a 'Date' header. Authorization=Basic <<<<<<<>>>>>>>>>=="}]
18/02/15 08:54:29 DEBUG HttpMethodBase: re-creating response stream from byte array
18/02/15 08:54:29 DEBUG HttpMethodBase: re-creating response stream from byte array
18/02/15 08:54:29 DEBUG HttpMethodBase: re-creating response stream from byte array
18/02/15 08:54:29 DEBUG HttpMethodBase: re-creating response stream from byte array
18/02/15 08:54:29 DEBUG HttpMethodBase: re-creating response stream from byte array
18/02/15 08:54:29 DEBUG HttpMethodBase: re-creating response stream from byte array
18/02/15 08:54:29 DEBUG HttpMethodBase: re-creating response stream from byte array
18/02/15 08:54:29 DEBUG HttpMethodBase: re-creating response stream from byte array
18/02/15 08:54:29 TRACE CommonsHttpTransport: Closing HTTP transport to https://<<<<<<<>>>>>>>>>
18/02/15 08:54:29 TRACE HttpConnection: enter HttpConnection.close()
18/02/15 08:54:29 TRACE HttpConnection: enter HttpConnection.closeSockedAndStreams()
18/02/15 08:54:29 TRACE CommonsHttpTransport: Closing HTTP transport to https://<<<<<<<>>>>>>>>>
18/02/15 08:54:29 TRACE HttpConnection: enter HttpConnection.close()
18/02/15 08:54:29 TRACE HttpConnection: enter HttpConnection.closeSockedAndStreams()
18/02/15 08:54:29 TRACE CommonsHttpTransport: Closing HTTP transport to https://<<<<<<<>>>>>>>>>
18/02/15 08:54:29 TRACE HttpConnection: enter HttpConnection.close()
18/02/15 08:54:29 TRACE HttpConnection: enter HttpConnection.closeSockedAndStreams()
18/02/15 08:54:29 TRACE CommonsHttpTransport: Closing HTTP transport to https://<<<<<<<>>>>>>>>>
18/02/15 08:54:29 TRACE HttpConnection: enter HttpConnection.close()
18/02/15 08:54:29 TRACE HttpConnection: enter HttpConnection.closeSockedAndStreams()
18/02/15 08:54:29 ERROR Executor: Exception in task 1.0 in stage 0.0 (TID 1)
org.elasticsearch.hadoop.EsHadoopIllegalArgumentException: Cannot detect ES version - typically this happens if the network/Elasticsearch cluster is not accessible or when targeting a WAN/Cloud instance without the proper setting 'es.nodes.wan.only'
	at org.elasticsearch.hadoop.rest.InitializationUtils.discoverEsVersion(InitializationUtils.java:190)

Is the authorization format wrong here?