Spark cannot connect to Elastic Cloud with Shield enabled

Elastic Stack Elasticsearch
1

Hi all,
I tried to connect my Spark code with Elastic Cloud without success. Following this post https://www.elastic.co/guide/en/cloud/current/shield.html, I have a configuration as follow:
es.http.timeout=5m
es.index.auto.create=true
es.net.http.auth.pass= mypass
es.net.http.auth.user= myuser
es.net.ssl=true
es.net.ssl.cert.allow.self.signed=true
es.net.ssl.protocol=true
es.nodes=46****.us-east-1.aws.found.io
es.nodes.wan.only=true
es.port=9343
es.scroll.size=50

  • Im using Spark 1.6.0, trial version of Elastic Cloud - Elasticsearch 2.2.0, es-hadoop connector elasticsearch-spark_2.10-2.2.0-rc1 => with this setup I can connect to AWS Elasticsearch cluster ( without authentication).
  • Here is the error with above configuration:
    org.elasticsearch.hadoop.EsHadoopIllegalStateException: Cannot instantiate SSL
  • Here is the error without SSL enable:
    org.elasticsearch.hadoop.rest.EsHadoopNoNodesLeftException: Connection error (check network and/or proxy settings)- all nodes failed; tried [[23.23.98.74:9343]]
  • I have tried with different port: 9200, 9243, 9343, 80, 443 with and without SSL enable but cannot connect. Could you please point out the configuration to connect from Spark to Elastic Cloud with Shield. Any help is appreciate !!
2

Where do folks come up with this setting? It's the second time this month and I have no idea where folks copy this incorrect configuration; in case the name is not clear enough, the docs clearly state:

SSL protocol to be used

"true" is not a valid protocol - anything wrong with the default "TLS"?

Instead of overriding the defaults (es.scroll.size or the timeout), reading the docs and only configuring the nodes and maybe the port is a much better time investment,.

3

Thanks a lot for your help, Costin ! I have kept the es.net.ssl.protocol as default and tried with different port configurations : 9200, 9243, 9343, 443, 80 with no success. I have read through : Apache Hadoop with Shield , es-hadoop configuration and es-hadoop security but still cannot figure out what should be the right configuration :confused: . Tried to find in the documentation of Elastic Cloud as well but the closest configuration available is : Java Transport Client . Do u have any experience connecting es-hadoop with Elastic Cloud ( Found) or any configuration could be derive similarly to the Java Transport Client ?

4

What is the error that you are getting? And what is your ES-Cloud configuration?
If you need help with the cloud setup, I recommend the dedicated forum here.