Fail to connect spark-elasticsearch with SSL

Elastic Stack Elasticsearch
1

Hi, I'm trying to connect spark with elasticsearch using the elastic-spark connector (https://www.elastic.co/guide/en/elasticsearch/hadoop/master/spark.html). Elastic is located on a different machine from which I launch the spark-submit command and it is configured using SSL protocol.

The configuration I am using is as follows:

val config= new SparkConf()
.set( "spark.es.net.http.auth.user" ,args(0))
.set( "spark.es.net.http.auth.pass" ,args(1))
.set( "es.nodes" , nodes)
.set( "es.port" , port)
.set( "es.nodes.wan.only" , "false" )
.set( "es.net.ssl" , "true" )
.set( "es.net.ssl.keystore.location" , "hdfs:///user/uappdaml/elasticsearch.keystore")

I generated the keystore with the following elastic command:

bin/elasticsearch-keystore create

I have also tried putting the keystore locally instead of hdfs to see if that was the problem, but still giving the same error. The error is as follows:

19/11/06 15:42:57 ERROR yarn.ApplicationMaster: User class threw exception: org.elasticsearch.hadoop.EsHadoopIllegalArgumentException: Cannot detect ES version - typically this happens if the network/Elasticsearch cluster is not accessible or when targeting a WAN/Cloud instance without the proper setting 'es.nodes.wan.only'

org.elasticsearch.hadoop.EsHadoopIllegalArgumentException: Cannot detect ES version - typically this happens if the network/Elasticsearch cluster is not accessible or when targeting a WAN/Cloud instance without the proper setting 'es.nodes.wan.only'

at org.elasticsearch.hadoop.rest.InitializationUtils.discoverClusterInfo(InitializationUtils.java:340)

at org.elasticsearch.spark.sql.EsSparkSQL$.saveToEs(EsSparkSQL.scala:76)

at org.elasticsearch.spark.sql.EsSparkSQL$.saveToEs(EsSparkSQL.scala:63)

at org.elasticsearch.spark.sql.package$SparkDataFrameFunctions.saveToEs(package.scala:43)

at com.investigationCenter.paymentsToElastic$.sameContext(paymentsToElastic.scala:53)

at com.investigationCenter.paymentsToElastic$.main(paymentsToElastic.scala:18)

at com.investigationCenter.paymentsToElastic.main(paymentsToElastic.scala)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:498)

at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:552)

Caused by: org.elasticsearch.hadoop.EsHadoopIllegalStateException: Cannot initialize SSL - Expected to find keystore file at [hdfs:///user/uappdaml/elasticsearch.keystore] but was unable to. Make sure that it is available on the classpath, or if not, that you have specified a valid URI.

at org.elasticsearch.hadoop.rest.commonshttp.SSLSocketFactory.createSSLContext(SSLSocketFactory.java:175)

at org.elasticsearch.hadoop.rest.commonshttp.SSLSocketFactory.getSSLContext(SSLSocketFactory.java:160)

at org.elasticsearch.hadoop.rest.commonshttp.SSLSocketFactory.createSocket(SSLSocketFactory.java:129)

at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707)

at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387)

at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171)

at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)

at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323)

at org.elasticsearch.hadoop.rest.commonshttp.CommonsHttpTransport.doExecute(CommonsHttpTransport.java:685)

at org.elasticsearch.hadoop.rest.commonshttp.CommonsHttpTransport.execute(CommonsHttpTransport.java:664)

at org.elasticsearch.hadoop.rest.NetworkClient.execute(NetworkClient.java:116)

at org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:432)

at org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:428)

at org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:388)

at org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:392)

at org.elasticsearch.hadoop.rest.RestClient.get(RestClient.java:168)

at org.elasticsearch.hadoop.rest.RestClient.mainInfo(RestClient.java:745)

at org.elasticsearch.hadoop.rest.InitializationUtils.discoverClusterInfo(InitializationUtils.java:330)

... 11 more

Caused by: org.elasticsearch.hadoop.EsHadoopIllegalArgumentException: Expected to find keystore file at [hdfs:///user/uappdaml/elasticsearch.keystore] but was unable to. Make sure that it is available on the classpath, or if not, that you have specified a valid URI.

at org.elasticsearch.hadoop.rest.commonshttp.SSLSocketFactory.loadKeyStore(SSLSocketFactory.java:195)

at org.elasticsearch.hadoop.rest.commonshttp.SSLSocketFactory.loadKeyManagers(SSLSocketFactory.java:215)

at org.elasticsearch.hadoop.rest.commonshttp.SSLSocketFactory.createSSLContext(SSLSocketFactory.java:173)

... 28 more

The command I use to launch the process is as follows:

spark-submit --master yarn --num-executors 1 --executor-cores 1 --executor-memory 1g --files hdfs:///user/uappdaml/application.properties --jars hdfs:///user/uappdaml/elasticsearch-spark-13_2.10-7.4.0.jar --class myclass --deploy-mode cluster hdfs:///user/uappdaml/myJar.jar user password

The machines have visibility because I can do curls and pings from spark machine to elastic machine.
I am not an expert in the subject of certificates but the keystore path seems to fail, does anyone know what could be happening?

A greeting and thanks in advance.

2

The "elasticsearch.keystore" is not an SSL keystore. Despite the confusing name, they're very different things.

The keystore you want for SSL is a JKS or PKCS#12 (.p12) file.
If you provide your Elasticsearch config file, we can probably point you to what you need.

3

Thanks for the reply.
I tried with jks and p12 files but the same error follows.
I have requested access to the elasticsearch.yml file when I get it updated.
Regards.

4

Hi again I get the elastic configuration:

xpack.security.enabled: true
bootstrap.system_call_filter: false
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: /software/ELK-7.4/ELK-7.4.2/elasticsearch-7.4.2/config/elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: /software/ELK-7.4/ELK-7.4.2/elasticsearch-7.4.2/config/elastic-certificates.p12
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.keystore.path: /software/ELK-7.4/ELK-7.4.2/elasticsearch-7.4.2/config/elastic-certificates.p12
xpack.security.http.ssl.truststore.path: /software/ELK-7.4/ELK-7.4.2/elasticsearch-7.4.2/config/elastic-certificates.p12
xpack.security.http.ssl.client_authentication: optional

The problem has changed but keep failing along the path:

Caused by: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
        at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
        at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1949)
        at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:302)
        at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:296)
        at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1514)

How can I generate keystore file and truststore file with elasticsearch?

5

This file contains the CA that you want to use from spark.
You probably don't want to use that file as-is because it contains the private key(s) for your cluster, and you don't need to give them to spark.

You can use keytool to export the CA from that keystore.

keytool -importkeystore \
  -srckeystore ./config/elastic-certificates.p12 -srcstorepass "" -srcalias ca \
  -destkeystore ./ca.p12
6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.