# Apache Spark to query Elasticsearch (https and basic authentication) **URL:** \ **Category:** Elasticsearch\ **Tags:** es-hadoop\ **Created:** [August 28, 2019, 4:52am UTC](https://discuss.elastic.co/t/apache-spark-to-query-elasticsearch-https-and-basic-authentication/197027 "2019-08-28T04:52:39Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![Darshan\_Parab](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/darshan_parab/32/51098_2.png) [@Darshan\_Parab](https://discuss.elastic.co/u/Darshan_Parab)\ **Post date:** [August 28, 2019, 4:52am UTC](https://discuss.elastic.co/t/apache-spark-to-query-elasticsearch-https-and-basic-authentication/197027/1 "2019-08-28T04:52:39Z")
Use case: Query secure Elasticsearch cluster (https and basic authentication enabled) using Apache Spark(pyspark and spark-submit) What I tried: start pyspark as follows: `./bin/pyspark --jars ./jars/elasticsearch-hadoop-7.2.0.jar --files /opt/ssl/jkeystore/elastic --driver-class-path /opt/ssl/jkeystore/elastic --conf "spark.executor.extraJavaOptions=-Djavax.net.ssl.trustStore=elastic" --conf "spark.execurot.extraJavaOptions=-Djavax.net.ssl.trustStorePassword=xxxxxx"` Query Elasticsearch as follows: `df = spark.read.format("org.elasticsearch.spark.sql").option("es.nodes","https://elasticsearch:9200").option("es.resource","index/_doc").option("es.read.field.as.array.include","tags").option("es.net.http.auth.user","user").option("es.net.http.auth.pass","password").option("es.net.ssl","true").load()` I'm getting error as below: `Caused by: org.elasticsearch.hadoop.rest.EsHadoopTransportException: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target` `Caused by: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target` `Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target` Apparently it looks like spark is unable to understand truststore settings. Elasticsearch Hadoop doesn't have any options to add certificates to keystore file using [secure settings](https://www.elastic.co/guide/en/elasticsearch/hadoop/current/security.html#keystore). How do I configure it correctly so that I can talk to Elasticsearch? ---
**Author:** ![james.baiera](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/james.baiera/32/10209_2.png) [@james.baiera](https://discuss.elastic.co/u/james.baiera)\ **Post date:** [September 6, 2019, 4:12pm UTC](https://discuss.elastic.co/t/apache-spark-to-query-elasticsearch-https-and-basic-authentication/197027/2 "2019-09-06T16:12:44Z")
The secure settings in ES-Hadoop are just for storing password configurations so they are not in the job configuration as plaintext. ES-Hadoop does support reading truststore and keystore files using these [SSL Settings](https://www.elastic.co/guide/en/elasticsearch/hadoop/current/security.html#_ssltls_configuration). It's important to also note, that when specifying truststores or keystores, the files you are referencing are available on the classpath of the driver and worker processes and thus looked up by name, or are available on every node's local filesystem in the same location (make sure to use the `file:///full/path/to/keystore` format instead of just the path) ---
**Author:** ![Darshan\_Parab](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/darshan_parab/32/51098_2.png) [@Darshan\_Parab](https://discuss.elastic.co/u/Darshan_Parab)\ **Post date:** [September 25, 2019, 11:57am UTC](https://discuss.elastic.co/t/apache-spark-to-query-elasticsearch-https-and-basic-authentication/197027/3 "2019-09-25T11:57:29Z")
So to communicate with SSL enables elasticsearch URL using pyspark? In elasticsearch library for python, it has option to pass the certificate. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [October 23, 2019, 11:57am UTC](https://discuss.elastic.co/t/apache-spark-to-query-elasticsearch-https-and-basic-authentication/197027/4 "2019-10-23T11:57:34Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.