Testing TLS/SSL secured elastic cluster

In this config (and earlier in this thread) you have

xpack.security.transport.ssl.enabled: true

twice, but do not have

xpack.security.http.ssl.enabled: true

If you want the node to be accessible over https, then you must enable http.ssl

If you want one single certificate for the whole cluster, then you do not want a certificate per node and should answer N to this question.

Yes, your certs/elastic-certificates.p12 file is intended to secure the transport protocol that the nodes use to connect to each other. For http you should use the http.p12 you generated.

You should enter every hostname/IP that any client (including curl, or administrative tools) will ever use to connect to your cluster. If in doubt, that should be every node.

just master nodes and hosts which will be used by java app client to request the cluster via https

It is not recommended to have clients connect directly to master nodes. If your cluster has dedicated master nodes, then they should be dedicated to being master nodes, and should not handle client requests.

Thanks for the quick response Tim.
That twice repeated line in configuration yml file is just typo error here, apology for the inconvenience. In the real cluster it was the correct line.
elasticsearch.yml

... cluster name, node name, etc, ...
# Security configuration
xpack.security.enabled: true
# TLS/SSL encryption inter-node communication
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: certs/elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: certs/elastic-certificates.p12
# TLS/SSL encryption http client communication
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.keystore.path: "http.p12"
xpack.security.http.ssl.truststore.path: "http.p12"
xpack.security.http.ssl.client_authentication: optional
xpack.security.transport.ssl.supported_protocols: [ "TLSv1.2", "TLSv1.1", "TLSv1"]

I re-generated certificates for inter-node and http client communication and was able to test different use cases with the curl and authorized user as follows

curl --cacert /path/to/http.crt -u estester:estester_pwd -XGET 'https://ip:port/_cat/nodes?v'
curl --cacert /path/to/http.crt -u estester:estester_pwd -XGET 'https://host:port/_cat/nodes?v'
curl -u estester:estester_pwd -XGET 'https://host:port/_cat/nodes?v'
curl --cacert /path/to/http.crt -XGET 'https://host:port/_cat/nodes?v'

etc.

Thank you all - Kavier, Ioannis, Albert Zaharovits and Tim Vernum - for your patience, help and generosity.

Now, when the x-pack secured cluster is configured and tested with the curl, I'm going to implement java app to test it with elastic cluster.

If you can reference me to simple java code samples to test secured elastic cluster with the client certificate and authorized user via https, it will be appreciated.

Running java client on Linux gives ClassNotFoundException
I implemented simple java client based on JEST API
Project contains jest-6.3.1.jar and x-pack-transport-7.8.1.jar
Compiled java code successfully, ran it on Linux and got the exception
Details are below

import java.io.InputStream;
import java.nio.file.Files;
import java.nio.file.Path;
import java.security.KeyStore;
import java.security.cert.Certificate;
import java.security.cert.CertificateFacrory;
import java.util.HashMap;
import java.util.Map;

import javax.net.ssl.HostnameVerifier;
import javax.net.ssl.SSLContext;

import org.apache.http.conn.ssl.NoopHostnameVerifier;
import org.apache.http.conn.ssl.SSLConnectionSocketFactory;
import org.apache.http.nio.conn.SchemeIOSessionStrategy;
import org.apache.http.nio.conn.SSLIOSessionStrategy;
import org.apache.http.ssl.SSLContextBuilder;
import org.apache.http.ssl.SSLContexts;

import com.google.gson.JsonObject;
import io.searchbox.client.JestClient;
import io.searchbox.client.JestClientFactory;
import io.searchbox.client.JestResult;
import io.searchbox.client.config.HttpClientConfig;
import io.searchbox.core.Search;

public class JestConnSecurES {
	public static void main() throws Exception {

String host=args[0];
String port=args[1];
String index=args[2];
String path_to_ca_crt=args[3];
String accessToken=args[4];

Path trustStorePath = Paths.get("/path/to/truststore.p12");
KeyStore truststore = KeyStore.getInstance("pkcs12");
try (InputStream is = Files.newInputStream(trustStorePath)) {
    truststore.load(is, keyStorePass.toCharArray());
}
SSLContextBuilder sslBuilder = SSLContexts.custom()
    .loadTrustMaterial(truststore, null);
final SSLContext sslContext = sslBuilder.build();

//CA certificate is available as PEM encoded file *.crt
Path caCrtificatePath=Path.get(path_to_ca_crt);
CertificateFactory certFactory=
		CertificateFactory.getInstance("X.509");
Certificate trustedCa;
try (InputStream is=Files.newInputStream(caCrtificatePath=Path)){
	trustedCa=certFactory.generateCertificate(is);
}

KeyStore trustStore=keyStore.getInstance("pkcs12");
trustStore.load(null,null);
trustStore.setCertificateEntry("ca", trustedCa);
SSLContextBuilder sslContextBuilder=SSLContexts.custom()
	.loadTrustMaterial(trustStore, null);
final SSLContext sslContext=sslContextBuilder.build();

HostnameVerifier hostnameVerifier=NoopHostnameVerifier.INSTANCE;

SSLConnectionSocketFactory sslSocketFactory=
	new SSLConnectionSocketFactory(sslContext, hostnameVerifier);
SchemeIOSessionStrategy httpsIOSessionStrategy=
	new SSLIOSessionStrategy(sslContext, hostnameVerifier);

JestClientFactory factory=new JestClientFactory();
factory.setHTTPClientConfig(new HttpClientConfig.Builder("https://"+host+":"+port)
	.defaultSchemeForDiscoveredNodes("https")
	.sslSocketFactory(sslSocketFactory)
	.httpsIOSessionStrategy(httpsIOSessionStrategy)
	.build()
);

JestClient client=factory.getObject();
String query="";
Search search=null;

Map<String,Object> header=new HashMap<>;
header.put("Authorization", "Basic "+accessToken);
search=new Search.Builder(query).addIndex.setHeader(header).build();

JestResultn result=client.execute(search);
JsonObject jo=result.getJsonObject();
System.out.println("search result \n "+jo.toString());
	}
}

Running java code on Linux
java -jar [java_main_class] [host] [port] [index] [path_to_ca_crt] [accessToken]

Console output

Error: A JNI error has occurred, please check your installation and try again
    Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/http/conn/socket/LayeredConnectionSocketFactory
    .......................................................................................................... 
    Caused by: java.lang.ClassNotFoundException: org.apache.http.conn.socket.LayeredConnectionSocketFactory

Am I missing some specific dependency, jar(s) ?
Any help will be appreciated

The code you shared has nothing Elasticsearch specific in it, so it's hard for us to troubleshoot what might be wrong.

Instead of treating ES as a generic REST API that you interact over HTTP , you can also use the High Level Rest Client, details are available in our docs: https://www.elastic.co/guide/en/elasticsearch/client/java-rest/current/java-rest-high.html

Thanks for the response Ioannis

In the shared code I tried to achieve:

[1]using JEST API, because there are lots of legacy code written on it for java client, currently communicating with ES-2.3/ES-5.6 clusters with ReadOnlyRest plugin and simplify the migration to newest ES-7.8
[2]use X-Pack security instead of ROR plugin for java client to communicate with secured ES-7.8

Q1: In your opinion, is it possible to use JEST API to communicate with X-Pack secured ES-7.8 cluster or the only High or Low Level Rest Clients should be used without mixing with JEST API ?

Q2: Which one is more preferable or must be used for java client implementation - High or Low Level Rest Client ?

Some explanation to shared code and issues:

public class JestConnSecurES {
	public static void main() throws Exception {

String host=args[0];
String port=args[1];
String index=args[2];
String path_to_ca_crt=args[3];
String accessToken=args[4];
//CA certificate is available as PEM encoded file *.crt
Path caCrtificatePath=Path.get(path_to_ca_crt);
CertificateFactory certFactory=CertificateFactory.getInstance("X.509");
Certificate trustedCa;
      try (InputStream is=Files.newInputStream(caCrtificatePath=Path)){
            trustedCa=certFactory.generateCertificate(is);
      }

KeyStore trustStore=keyStore.getInstance("pkcs12");
trustStore.load(null,null);
trustStore.setCertificateEntry("ca", trustedCa);
SSLContextBuilder sslContextBuilder=SSLContexts.custom()
      .loadTrustMaterial(trustStore, null);
final SSLContext sslContext=sslContextBuilder.build();

HostnameVerifier hostnameVerifier=NoopHostnameVerifier.INSTANCE;

SSLConnectionSocketFactory sslSocketFactory=
     new SSLConnectionSocketFactory(sslContext, hostnameVerifier);
SchemeIOSessionStrategy httpsIOSessionStrategy=
    new SSLIOSessionStrategy(sslContext, hostnameVerifier);

JestClientFactory factory=new JestClientFactory();
factory.setHTTPClientConfig(new 
    httpClientConfig.Builder("https://"+host+":"+port)
    .defaultSchemeForDiscoveredNodes("https")
    .sslSocketFactory(sslSocketFactory)
    .httpsIOSessionStrategy(httpsIOSessionStrategy)
    .build() );

JestClient client=factory.getObject();
String query="";
Search search=null;

Map<String,Object> header=new HashMap<>;
header.put("Authorization", "Basic "+accessToken);
search=new Search.Builder(query).addIndex.setHeader(header).build();

JestResultn result=client.execute(search);
JsonObject jo=result.getJsonObject();
System.out.println("search result \n "+jo.toString());
    }
}

[3]java client project in Eclipse/pom.xml references jest-6.3.1.jar and x-pack-transport-7.8.1.jar as advised at https://www.elastic.co/guide/en/elasticsearch/reference/current/setup-xpack-client.html
[4]first section of code from "Path trustStorePath = " till "final SSLContext sslContext=" I got as it is at https://www.elastic.co/guide/en/elasticsearch/client/java-rest/7.9/_encrypted_communication.html
because it allows to use the certificate file *.crt I tested with curl successfully on the cluster
In this section the "org.apache.http.conn.ssl.SSLContextBuilder" and "org.apache.http.conn.ssl.SSLContexts" say "SSLContextBuilder and SSLContexts deprecated"

Q3: Which packages/jars do contain these deprecated SSLContextBuilder and SSLContexts ?

[5]section of code from "JestClient client=factory.getObject()" till "JsonObject jo=result.getJsonObject()" has been tested successfully previously with ES-5.6/ROR plugin cluster
[6]shared java code constructs trustStorePath, sslBuilder, sslContext, caCrtificatePath, trustStore, sslContextBuilder, sslContext, sslSocketFactory, factory, client, header, search, result with requested index
The code accepts input arguments: host, port, index to search, path to certificate

Q4: based on [3]-[6] I'm not clear why you're saying "code you shared has nothing Elasticsearch specific in it" ?
I might be missing something, having not much experience in the field

I hope having answers to Q1-Q4 and references to practical examples can help me to proceed further in right direction with java client implementation.
Also will follow up your advise to look at High Level Rest Client

Many thanks in advance

Hi,

Apologies for the previous post, I didn't mean to put you off - sorry if it came out this way.

I have no working knowledge of this JEST API that you mention so I can't state an opinion. I know for a fact that {H,L}LRC work fine and that the transport client is deprecated in 7 and will be removed in 8 so my suggestion is to use the rest client.

I would use the HLRC unless you know why you have a specific reason to use the LLRC instead.

It's in the apache http client.

Look at your imports. You are not using anything from the elasticsearch transport client in your code, and there is nothing elasticsearch related in your code. You use some libraries to make a call to an HTTP REST API ( that, ok, happens to be elasticsearch in this case , but this is not relevant in that context )

In 3 you just added the xpack client as a dependency but your are not using it anywhere.

In 4 you are copying part of an example from a LLRC doc and attempt to apply it to a transport client setup, which won't work.

It's really hard for folks to provide generic consulting on development efforts in such kind of forums. I'd start by reading the rest client docs that were linked above. They have all the necessary setup and they contain examples and sections to get you going.

HTH

Thank you Ioannis for your attention and detailed answers, will read HLRC docs and give it a try.

np, hope this helps you further!

If you have specific questions, issues while trying to use the rest client, folks in the forums will do their best to help out.

I implemented and successfully tested java JEST client as follows

pom.xml

<dependency>
   <groupId>io.searchbox</groupId>
   <artifactId>jest</artifactId>
   <version>6.3.1</version>
</dependency>
<dependency>
   <groupId>io.searchbox</groupId>
   <artifactId>jest-common</artifactId>
   <version>6.3.1</version>
</dependency>

Source code of project JestToSecurES in Eclipse (Neon)

import java.io.InputStream;
import java.nio.file.Files;
import java.nio.file.Path;
import java.security.KeyStore;
import java.security.cert.CertificateFacrory;
import java.util.HashMap;
import java.util.Map;

import javax.net.ssl.HostnameVerifier;

import org.apache.http.conn.ssl.NoopHostnameVerifier;
import org.apache.http.conn.ssl.SSLConnectionSocketFactory;
import org.apache.http.nio.conn.SchemeIOSessionStrategy;
import org.apache.http.nio.conn.SSLIOSessionStrategy;
import org.apache.http.ssl.SSLContextBuilder;
import org.apache.http.ssl.SSLContexts;

import com.google.gson.JsonObject;

import io.searchbox.client.JestClient;
import io.searchbox.client.JestClientFactory;
import io.searchbox.client.JestResult;
import io.searchbox.client.config.HttpClientConfig;
import io.searchbox.core.Search;

public class JestConnSecurES {
	public static void main() throws Exception {

System.out.println("Running main class JestConnSecurES of project JestToSecurES);	
System.out.println("Input arguments: "+"\n host: "+host"+"\n port: "+port+"\n index: "+index+"\n path_to_ca_crt: "+path_to_ca_crt+"\n accessToken: "+accessToken);
	
String host=args[0];
String port=args[1];
String index=args[2];
String path_to_ca_crt=args[3];
String accessToken=args[4];

Path caCrtificatePath=Path.get(path_to_ca_crt);
CertificateFactory certFactory=
	CertificateFactory.getInstance("X.509");
java.security.cert.Certificate trustedCa;
try (InputStream is=Files.newInputStream(caCrtificatePath)){
	trustedCa=certFactory.generateCertificate(is);
}

KeyStore trustStore=keyStore.getInstance("pkcs12");
trustStore.load(null,null);
trustStore.setCertificateEntry("ca", trustedCa);
SSLContextBuilder sslContextBuilder=SSLContexts.custom()
	.loadTrustMaterial(trustStore, null);
final javax.net.ssl.SSLContext sslContext=sslContextBuilder.build();
	
HostnameVerifier hostnameVerifier=NoopHostnameVerifier.INSTANCE;

SSLConnectionSocketFactory sslSocketFactory=
	new SSLConnectionSocketFactory(sslContext, hostnameVerifier);
SchemeIOSessionStrategy httpsIOSessionStrategy=
	new SSLIOSessionStrategy(sslContext, hostnameVerifier);

JestClientFactory factory=new JestClientFactory();
factory.setHttpClientConfig(new HttpClientConfig.Builder("https://"+host+":"+port)
	.defaultSchemeForDiscoveredNodes("https")
	.sslSocketFactory(sslSocketFactory)
	.httpsIOSessionStrategy(httpsIOSessionStrategy)
	.build()
);

JestClient client=factory.getObject();
String query="";
Search search=null;

Map<String,Object> header=new HashMap<>;
header.put("Authorization", "Basic "+accessToken);
search=new Search.Builder(query).addIndex.setHeader(header).build();

JestResult result=client.execute(search);
JsonObject jo=result.getJsonObject();
System.out.println("search result \n "+jo.toString());
	}
}

Running project as Maven build and goal assempbly:single will produce JestToSecurES.jar with main class JestConnSecurES in its MANIFEST.MF file

Running/Testing java client on Linux

Notes:
-Place http.crt (mentioned above in this post) along with JestToSecurES.jar in particular directory
-Create/generate value of accessToken by encoding the value of auth_user:auth_user_password in Base64
-The index books has been initially created in the cluster and as previously noted the authenticated user has been granted permissions to monitor the cluster and all operations on indices

Allowed accessToken
java -jar JestToSecurES.jar [host_or_ip] [port] books http.crt [Base64(auth_user:auth_user_password)]
Console output will contain: ... "total":1,"successfull":1 ... "_index":"books","type":"book" ...

Forbidden accessToken (change previous value of [Base64(auth_user:auth_user_password)] somehow to corrupt it)
java -jar JestToSecurES.jar [host_or_ip] [port] books http.crt [Base64(auth_user:auth_user_password)]
Console output will contain: ... "error":{"root_cause"} ... "security_exception" ... "unable to authenticate user" ... for REST request [/books/_search]...

Goals:
-setup/configure fully secured elastic cluster with X-Pack (TLS/SSL for internal and external communication) on RHEL
-implement RBAC for authenticated user
-implement simple java client based on JEST API and test it with secured cluster

Summary of proceeded steps to achieve the goals
[1]Setup and configured 2 host (two nodes on each) ES-7.8.0 unsecured cluster on Linux
-Created index books
-Tested successfully unsecured cluster with curl
[2]Enabled x-pack on the cluster and configured TLS/SSL for internal communication between nodes (elasticsearch.yml)
-Implemented Role Based Access Control
--Created authenticated user, user password via elastic api commands (not using Kibana), created particular role
--Granted permissions to particular role and assigned the authenticated user for this role
-Tested successfully cluster with curl to make sure RBAC works as expected
[3]Configured TLS/SSL on HTTP to being able to make request with httpS
--changed elasticsearch.yml accordingly
--generated elastic-certificate.p12 with bin/elasticsearch-certutil
--extracted certificate from elastic-certificate.p12 into http.p12 with openssl
-Tested successfully fully secured cluster with curl
[4]Created simple java app based on JEST API
-Tested successfully fully secured elastic cluster with simple java client

To implement java client on JEST, reading elastic doc HLRC/LLRC allowed me to understand how to incorporate http.crt into java jest client.
I'll try to implement java client on HLRC/LLRC later on.

I agree with you, it is not easy to consult developers on this topic with different issues, because there are lots of dependencies like releases, used libs, system platforms, developers work on, different terminologies/meanings used for the same things or vice versa, etc., regardless elastic/x-pack doc is very detailed and contains lots of cross-references.

I think having simple use case sample for HLRC/LLRC java client with the steps I proceeded above from end-to-end, will allow developers to proceed with everything efficient and make the process more understandable and enjoyable.

Thank you all, elastic team for sharing your experience/knowledge globally and your help and support.

Next task is to make hadoop-elasticsearch/connector works with this TLS/SSL x-pack secured elastic cluster.

ES-Hive integration is not working on secured elastic cluster

The details are below

Elasticsearch 7.8.0 (basic license), Hive 1.2.1000.2.6.5.0-292
TLS/SSL secured cluster has been configured and tested successfully with curl command and java JEST client on RHEL

curl --cacert /path/to/http.crt -u estester:estester_pwd -XGET 'https://ip:port/_cat/nodes?v'
curl --cacert /path/to/http.crt -u estester:estester_pwd -XGET 'https://ip:port/_cat/indices/books?pretty'

elasticsearch.yml

# Security configuration
xpack.security.enabled: true
# TLS/SSL encryption inter-node communication
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: certs/elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: certs/elastic-certificates.p12
# TLS/SSL encryption http client communication
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.keystore.path: "http.p12"
xpack.security.http.ssl.truststore.path: "http.p12"
xpack.security.http.ssl.client_authentication: optional
xpack.security.transport.ssl.supported_protocols: [ "TLSv1.2", "TLSv1.1", "TLSv1"]

Add jars

hive> ADD JAR hdfs://xxxxx/tmp/elasticsearch-hadoop-7.8.0.jar;
hive> ADD JAR .../commons-httpclient-3.0.1.jar;

Create table

hive> create external table if not exists db_name.tbl_name (
id string,
title string,
year string)
STORED BY 'org.elasticsearch.hadoop.hive.EsStorageHandler'
TBLPROPERTIES(
"es.nodes"="xx.xxx.xx.xxx",
"es.port"="xxxx",
"es.resource.read"="books/book",
"es.read.operation"="index",
"es.index.read.missing.as.empty"="true",
"es.nodes.discovery"="true",
"es.read.metadata"="true",
"es.mapping.names"="id:_metadata._id, title:title, year:year",
"es.net.ssl"="true",
"es.net.ssl.cert.allow.self.signed"="true",
"es.net.ssl.keystore.type"="PKCS12",
"es.net.ssl.keystore.location"="hdfs:///xxxxx_domain/tmp/http.p12",
"es.net.ssl.truststore.location"="hdfs:///xxxxx_domain/tmp/http.p12",
"es.net.http.auth.user"="estester",
"es.net.http.auth.pass"="estester_pwd");

Table created successfully, select failed with the details below

hive> select * from db_name.tbl_name;
OK
Failed with exception java.io.IOException:org.elasticsearch.hadoop.EsHadoopIllegalArgumentException: Cannot detect ES version- this typically happens if network/elasticsearch cluster is not accessible or when targetting a WAN/Cloud instance without proper setting in 'es.nodes.wan.only'

Note: When setting and configuring elastic cluster, I generated the following certificate files:
elastic-certificates.p12, elastic-stack-ca.p12, http.p12, http.crt
These files reside also in hdfs:///xxxxx/tmp/

A few questions please:
Q1: which of the above mentioned certificate file should be used for es.net.ssl.keystore.location and es.net.ssl.truststore.location ?

Q2: what would be the correct value for es.net.ssl.keystore.type ?

Q3: is referencing directory like hdfs:///xxxxx_domain/tmp correct for es.net.ssl.keystore.location and es.net.ssl.truststore.location ?

Am I missing something in table's properties or elsewhere ?

Thanks in advance

Based on the context that I've gathered from reading this thread, I think you'll want to set the truststore to the http.p12 file. Since Hive is acting as a client connection to the cluster, you should be able to ignore the keystore setting, similar to how you configured JEST up above.

keystore type you have is correct: PKCS12

Unfortunately, we do not support pulling keystore/truststore files from HDFS. You can deploy your truststore in one of two ways:

  1. Add the file to the job's classpath, and refer to it by name in the es.net.ssl.truststore.location property. ES-Hadoop will search the classpath for the resource and open it via the classpath. Since it is part of the job's classpath, Hive will make sure it is on every node.
  2. Arguably the easier choice if you have total control over your deployment, place the truststore on each node's local filesystem in your Hive cluster, at the same path location on each node. Refer to the file path using file:/// URL on the es.net.ssl.truststore.location setting. ES-Hadoop treats full file URI's as a local resource instead of one provided via the classpath.

A quick cursory check:

This isn't a valid property for ES-Hadoop. Read operations are always a scrolled search, this is not configurable.

I would avoid the dotted field name in the es.mapping.names property.

These settings can be provided in a keystore instead of in plaintext on the table properties if you would like: Security | Elasticsearch for Apache Hadoop [8.11] | Elastic

Hope that helps!

P.S. Just a quick clarification here: We've seen issues with Hive where adding a file to the Hive Classpath via the Hive settings is not sufficient to access it in ES-Hadoop. The way that we've found that works effectively is to use the ADD FILE HiveQL directive. I suggest just deploying the certificates to each node's local filesystem as a cleaner approach.

Thank you very much James
I'll follow up your guidelines and recommendations

The es-hadoop connector is not working yet
Here is what I did

Replaced these two entries in the above table properties

"es.net.ssl.keystore.location"="hdfs:///xxxxx_domain/tmp/http.p12",
"es.net.ssl.truststore.location"="hdfs:///xxxxx_domain/tmp/http.p12",

by one

"es.net.ssl.truststore.location"="file:///path_to_cert_on_local/http.p12",

Placed file http.p12 in the directory /path_to_cert_on_local/ on all 10 nodes/servers of Hadoop cluster

Started hive in debug mode
Added 2 jars: elasticsearch-hadoop-7.8.0.jar and commons-httpclient-3.0.1.jar
Added certificate file
ADD FILE /path_to_cert_on_local/http.p12

Executed create table and select from it

Log file shows
java.io.IOException:org.elasticsearch.hadoop.EsHadoopIllegalArgumentException: Cannot detect ES version- this typically happens if network/elasticsearch cluster is not accessible or when targetting a WAN/Cloud instance without proper setting in 'es.nodes.wan.only'
Caused by: org.elasticserch.hadoop.EsHadoopIllegalStateException: Cannot initialize SSL - Expected to find keystore file at [file:///path_to_cert_on_local/http.p12] but was unable to. Make sure that it is available on the classpath, or if not, that you have specified a valid URI.

The following entries in the table properties worked well without any issues when I tested es-hadoop connector with Non-secured es cluster

"es.read.operation"="index",
"es.mapping.names"="id:_metadata._id, title:title, year:year",

So, I'm still experiencing the issue.
Any recommendations will be appreciated.

I fixed the previous error by changing truststore location to

"es.net.ssl.truststore.location"="file:/path_to_cert_on_local/http.p12",`

Now when running sql select the LOG shows the following exception

... Open Connection to xx.xxx.xx.xxx:xxxx
...Loading keystore located at [file:/path_to_cert_on_local/http.p12]
...Content-Type: application/json
...Authorization: Basic LK...
...Usr-Agent: ...
...Host: xx.xxx.xx.xxx:xxxx
...Closing the connection     httpclient.HttpConnection.java.closeSocketAndStreams - Exception when closing output
javax.net.ssl.SSLException: Connection has been shutdown: javax.net.ssl.SSLException: java.lang.RuntimeException Unexpected error:
java.security.InvalidAlgorithmParameterException: 
the trustAnchors parameter must be non-empty

Anyone experienced and fixed this ?

It looks like your http.p12 is missing the trusted certificate entry that should have been generated by certutil.

What do you get from

keytool -list -keystore /path/to/http.p12 -storepass ""

(That's assuming your PKCS#12 file has no password, replace that final "" with the real password if you have one)

Thanks for the response Tim

When I generated the certificates with certutil for http.p12, I went with blank password option, simply pressed the "Enter" key at the prompt, as advised

The elasticsearch.yml (in this thread above) contains entries with certificates

# TLS/SSL encryption http client communication
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.keystore.path: "http.p12"
xpack.security.http.ssl.truststore.path: "http.p12"

and I tested the cluster already successfully with CURL and java jest client (in this thread above)

The output of

keytool -list -keystore /path/to/http.p12 -storepass ""

is as follows

Keystore type: PKCS12
Keystore provider: SUN

Your keystore contains 2 entries 

ca, Nov 3, 2020, trustedCertEntry,
Certificate fingerprint (SHA1) 46:E3 .................... E1:19
http, Oct 7, 2020, PrivateKeyEntry,
Certificate fingerprint (SHA1) 1A:C4 .................... 45:2G

Any recommendation how to make it work ?

I could Not make it work with the PKCS12 type in table properties like this

"es.net.ssl.keystore.type"="PKCS12",
"es.net.ssl.truststore.location"="file:/path_to_cert_on_local/http.p12",

How to make it work in es-hadoop v.7.8.0 with es v.7.8.0 ?
I didn't find any successful solution on the forum.

I made the following configuration work successfully with the JKS type:

Imported the http.crt into truststore.jks with the command

keytool -import -trustcacerts -alias tls -file http.crt -keystore truststore.jks

and the truststore.jks has been generated

hive> create external table if not exists db_name.tbl_name (
id string,
title string,
year string)
STORED BY 'org.elasticsearch.hadoop.hive.EsStorageHandler'
TBLPROPERTIES(
"es.nodes"="xx.xxx.xx.xxx",
"es.port"="xxxx",
"es.resource.read"="books/book",
"es.read.operation"="index",
"es.index.read.missing.as.empty"="true",
"es.nodes.discovery"="true",
"es.read.metadata"="true",
"es.mapping.names"="id:_metadata._id, title:title, year:year",
"es.net.ssl"="true",
"es.net.ssl.cert.allow.self.signed"="true",
"es.net.ssl.keystore.type"="jks",
"es.net.ssl.truststore.location"="file:/xxxxx_domain/tmp/truststore.jks",
"es.net.http.auth.user"="estester",
"es.net.http.auth.pass"="estester_pwd");

This configuration works well, I was able to run select from table successfully, but I'd like to take off the user/password in plain text from table properties and incorporate them into keystore file and reference this file in table properties

Importing user/password settings into keystore file - Not working
To import "es.net.http.auth.user"="estester" and "es.net.http.auth.pass"="estester_pwd" settings into keystore file I followed up

To create keystore file I ran the command as guided in the documentation

java -classpath elasticsearch-hadoop-7.8.0.jar org.elasticsearch.hadoop.cli.keytool create

and esh.keystore file has been created; than I ran

java -classpath elasticsearch-hadoop-7.8.0.jar org.elasticsearch.hadoop.cli.keytool add es.net.http.auth.user
enter value for es.net.http.auth.user:  [entered estester]
java -classpath elasticsearch-hadoop-7.8.0.jar org.elasticsearch.hadoop.cli.keytool add es.net.http.auth.pass
enter value for es.net.http.auth.pass:  [entered estester_pwd]

and in table properties I used

"es.net.ssl.keystore.location"="file:/xxxxx_domain/tmp/esh.keystore"

instead of "es.net.http.auth.user"="estester" and "es.net.http.auth.pass"="estester_pwd"

I added file

hive> ADD FILE /path_to_file/esh.keystore;

Recreated table, did select from table and log shows the Exception

.....
java.io.IOException:org.elasticsearch.hadoop.EsHadoopIllegalArgumentException: Cannot detect ES version- this typically happens if network/elasticsearch cluster is not accessible or when targetting a WAN/Cloud instance without proper setting in 'es.nodes.wan.only'
.....
Caused by: org.elasticsearch.hadoop.EsHadoopIllegalState exception: Cannot initialize SSL - Get Key failed: AES SecretKeyFactory not available
.....
Caused by: java.security.UnrecoverableKeyException: Get Key failed: AES SecretKeyFactory not available
.....
Caused by: java.security.NoSuchAlgorithmException: AES SecretKeyFactory not available

Q:How to create keystore file properly and import user/password into it ?

Q2: Which value should be assigned to "es.net.ssl.keystore.pass" and where it comes from if it is needed at all ?

es-hadoop with esh.keystore is not working

I tried also the following configuration to move the entry es.net.http.auth.pass and value estester_pwd from table to esh.keystore file

Table

..........
"es.net.ssl"="true",
"es.net.ssl.cert.allow.self.signed"="true",
"es.net.ssl.keystore.type"="jks",
"es.net.ssl.truststore.location"="file:/path_to_truststore/truststore.jks",
"es.net.http.auth.user"="estester",
"es.net.ssl.truststore.location"="file:/path_to_keystore/esh.keystore");

Added entry es.net.http.auth.pass and its value estester_pwd to the esh.keystore

Select from table failed and gives the same exception as before

..........
Caused by: java.security.NoSuchAlgorithmException: AES SecretKeyFactory not available
..........

I tried to add the entry es.net.http.auth.pass and its value estester_pwd to esh.keystore with double, single quotes and without quotes at all - nothing is working.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.