Spark cannot connect to Elastic Cloud with Shield enabled


(Pham Khanh) #1

Hi all,
I tried to connect my Spark code with Elastic Cloud without success. Following this post https://www.elastic.co/guide/en/cloud/current/shield.html, I have a configuration as follow:
es.http.timeout=5m
es.index.auto.create=true
es.net.http.auth.pass= mypass
es.net.http.auth.user= myuser
es.net.ssl=true
es.net.ssl.cert.allow.self.signed=true
es.net.ssl.protocol=true
es.nodes=46****.us-east-1.aws.found.io
es.nodes.wan.only=true
es.port=9343
es.scroll.size=50

  • Im using Spark 1.6.0, trial version of Elastic Cloud - Elasticsearch 2.2.0, es-hadoop connector elasticsearch-spark_2.10-2.2.0-rc1 => with this setup I can connect to AWS Elasticsearch cluster ( without authentication).
  • Here is the error with above configuration:
    org.elasticsearch.hadoop.EsHadoopIllegalStateException: Cannot instantiate SSL
  • Here is the error without SSL enable:
    org.elasticsearch.hadoop.rest.EsHadoopNoNodesLeftException: Connection error (check network and/or proxy settings)- all nodes failed; tried [[23.23.98.74:9343]]
  • I have tried with different port: 9200, 9243, 9343, 80, 443 with and without SSL enable but cannot connect. Could you please point out the configuration to connect from Spark to Elastic Cloud with Shield. Any help is appreciate !!

(Costin Leau) #2

Where do folks come up with this setting? It's the second time this month and I have no idea where folks copy this incorrect configuration; in case the name is not clear enough, the docs clearly state:

SSL protocol to be used

"true" is not a valid protocol - anything wrong with the default "TLS"?

Instead of overriding the defaults (es.scroll.size or the timeout), reading the docs and only configuring the nodes and maybe the port is a much better time investment,.


(Pham Khanh) #3

Thanks a lot for your help, Costin ! I have kept the es.net.ssl.protocol as default and tried with different port configurations : 9200, 9243, 9343, 443, 80 with no success. I have read through : Apache Hadoop with Shield , es-hadoop configuration and es-hadoop security but still cannot figure out what should be the right configuration :confused: . Tried to find in the documentation of Elastic Cloud as well but the closest configuration available is : Java Transport Client . Do u have any experience connecting es-hadoop with Elastic Cloud ( Found) or any configuration could be derive similarly to the Java Transport Client ?


(Costin Leau) #4

What is the error that you are getting? And what is your ES-Cloud configuration?
If you need help with the cloud setup, I recommend the dedicated forum here.


(system) #6