I am trying to connect to our production amazon ES cluster through a proxy machine, which only has permissions to the nodes to access es data.
PFB configs set,
sparkConf.set("spark.driver.allowMultipleContexts", "true")
sparkConf.set("es.index.auto.create", "true")
sparkConf.set("es.nodes.discovery", "true")
//sparkConf.set("es.nodes", "es_hostname:19200")
System.setProperty("http.proxyHost", "----");
System.setProperty("http.proxyPort", "9201");
sparkConf.set("es.nodes","----")
sparkConf.set("es.port","9200")
sparkConf.set("java.net.useSystemProxies","true")
So i have tried both ways of using proxy settings , other way as below,
sparkConf.set("es.net.proxy.https.host","----")
sparkConf.set("es.net.proxy.https.port","9201")
by using the first set of config the get the below exception,
Exception in thread "main" org.elasticsearch.hadoop.rest.EsHadoopInvalidRequest: [HEAD] on [logstash-listing_lifecycle_monitor-2016.06.14/] failed; server[<ip-of one of the machine in the es cluster,but not proxy hostname >:9200] returned [405|Method Not Allowed:]
not sure if this means that the request has been made to one of the nodes in the es cluster.
by settings as below,
sparkConf.set("es.net.proxy.https.host","----")
sparkConf.set("es.net.proxy.https.port","9201")
it tries connecting and throws connection exception.
Appreciate any kind of inpust on resolving the issue.