Continuing the discussion from Basic Authentication with Spark fails with 403(forbidden):
Hi
I have the same issue (maybe), I noticed i get the log messages
WARN HttpMethodDirector: Required credentials not available for BASIC <any realm>@localhost:8080
WARN HttpMethodDirector: Preemptive authentication requested but no default credentials available
Can you explain how to define these or do i have some other proxy issue.
SBT config:
scalaVersion := "2.10.6"
libraryDependencies += "org.apache.spark" % "spark-core_2.10" % "1.6.0"
libraryDependencies += "org.elasticsearch" % "elasticsearch-spark_2.10" % "2.1.0"
Spark Config:
val conf = new SparkConf().setAppName("Spark ElasticSearch")
conf.setMaster("local[1]")
.set("es.nodes", "localhost")
.set("es.port", "8080")
.set("es.nodes.discovery", "false")
//.set("es.net.http.auth.user", "username")
//.set("es.net.http.auth.pass", "password")
.set("es.net.proxy.http.host","proxy.host.com.au")
.set("es.net.proxy.http.port","8080")
.set("es.net.proxy.http.user","username")
.set("es.net.proxy.http.pass","password")
Log:
DefaultHttpParams: Set parameter http.useragent = Jakarta Commons-HttpClient/3.1
DefaultHttpParams: Set parameter http.protocol.version = HTTP/1.1
DefaultHttpParams: Set parameter http.connection-manager.class = class org.apache.commons.httpclient.SimpleHttpConnectionManager
DefaultHttpParams: Set parameter http.protocol.cookie-policy = default
DefaultHttpParams: Set parameter http.protocol.element-charset = US-ASCII
DefaultHttpParams: Set parameter http.protocol.content-charset = ISO-8859-1
DefaultHttpParams: Set parameter http.method.retry-handler = org.apache.commons.httpclient.DefaultHttpMethodRetryHandler@5c2fb6d8
DefaultHttpParams: Set parameter http.method.retry-handler = org.elasticsearch.hadoop.rest.commonshttp.CommonsHttpTransport$1@48ded677
DefaultHttpParams: Set parameter http.connection-manager.timeout = 60000
DefaultHttpParams: Set parameter http.socket.timeout = 60000
CommonsHttpTransport: Using authenticated HTTP proxy [proxy.host.com.au:8080]
HttpClient: Java version: 1.8.0_66
HttpClient: Java vendor: Oracle Corporation
HttpClient: Java class path: myclasspath
HttpClient: Operating system name: Windows 7
HttpClient: Operating system architecture: amd64
HttpClient: Operating system version: 6.1
HttpClient: SUN 1.8: SUN (DSA key/parameter generation;...
HttpClient: SunRsaSign 1.8: Sun RSA signature provider
HttpClient: SunEC 1.8: Sun Elliptic Curve provider (EC, ECDSA, ECDH)
HttpClient: SunJSSE 1.8: Sun JSSE provider(PKCS12..
HttpClient: SunJCE 1.8: SunJCE Provider (implements ..
HttpClient: SunJGSS 1.8: Sun (Kerberos v5, SPNEGO)
HttpClient: SunSASL 1.8: Sun SASL provider(implements client mechanisms for: DIGEST-MD5, GSSAPI,..
HttpClient: SunPCSC 1.8: Sun PC/SC provider
HttpClient: SunMSCAPI 1.8: Sun's Microsoft Crypto API provider
DefaultHttpParams: Set parameter http.authentication.preemptive = true
DefaultHttpParams: Set parameter http.tcp.nodelay = true
HttpMethodDirector: Preemptively sending default basic credentials
HttpMethodDirector: Authenticating with BASIC <any realm>@proxy.host.com.au:8080
HttpMethodParams: Credential charset not configured, using HTTP element charset
HttpMethodDirector: Authenticating with BASIC <any realm>@localhost:8080
WARN HttpMethodDirector: Required credentials not available for BASIC <any realm>@localhost:8080
WARN HttpMethodDirector: Preemptive authentication requested but no default credentials available
HttpConnection: Open connection to proxy.host.com.au:8080
header: >> "GET http://localhost:8080/ HTTP/1.1[\r][\n]"
HttpMethodBase: Adding Host request header
header: >> "Proxy-Authorization: Basic KioqKioqKio6KioqKioqKio=[\r][\n]"
header: >> "User-Agent: Jakarta Commons-HttpClient/3.1[\r][\n]"
header: >> "Host: localhost:8080[\r][\n]"
header: >> "Proxy-Connection: Keep-Alive[\r][\n]"
header: >> "[\r][\n]"
header: << "HTTP/1.1 403 Forbidden[\r][\n]"
header: << "HTTP/1.1 403 Forbidden[\r][\n]"
header: << "Content-Length: 0[\r][\n]"
header: << "Date: Wed, 27 Jan 2016 04:27:13 GMT[\r][\n]"
header: << "Via: HTTP/1.1 proxy10705, 1.1 proxy.host.com.au:3128 (Cisco-IronPort-WSA/7.5.1-201)[\r][\n]"
header: << "Connection: keep-alive[\r][\n]"
header: << "Proxy-Connection: keep-alive[\r][\n]"
header: << "[\r][\n]"
HttpMethodBase: Should NOT close connection in response to directive: keep-alive
HttpConnection: Releasing connection back to connection manager.
ERROR Executor: Exception in task 0.0 in stage 1.0 (TID 1)
org.elasticsearch.hadoop.rest.EsHadoopInvalidRequest: [GET] on [] failed; server[null] returned [403|Forbidden:]
at org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:336)
at org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:301)