np, hope this helps you further!
If you have specific questions, issues while trying to use the rest client, folks in the forums will do their best to help out.
I implemented and successfully tested java JEST client as follows
pom.xml
<dependency>
<groupId>io.searchbox</groupId>
<artifactId>jest</artifactId>
<version>6.3.1</version>
</dependency>
<dependency>
<groupId>io.searchbox</groupId>
<artifactId>jest-common</artifactId>
<version>6.3.1</version>
</dependency>
Source code of project JestToSecurES in Eclipse (Neon)
import java.io.InputStream;
import java.nio.file.Files;
import java.nio.file.Path;
import java.security.KeyStore;
import java.security.cert.CertificateFacrory;
import java.util.HashMap;
import java.util.Map;
import javax.net.ssl.HostnameVerifier;
import org.apache.http.conn.ssl.NoopHostnameVerifier;
import org.apache.http.conn.ssl.SSLConnectionSocketFactory;
import org.apache.http.nio.conn.SchemeIOSessionStrategy;
import org.apache.http.nio.conn.SSLIOSessionStrategy;
import org.apache.http.ssl.SSLContextBuilder;
import org.apache.http.ssl.SSLContexts;
import com.google.gson.JsonObject;
import io.searchbox.client.JestClient;
import io.searchbox.client.JestClientFactory;
import io.searchbox.client.JestResult;
import io.searchbox.client.config.HttpClientConfig;
import io.searchbox.core.Search;
public class JestConnSecurES {
public static void main() throws Exception {
System.out.println("Running main class JestConnSecurES of project JestToSecurES);
System.out.println("Input arguments: "+"\n host: "+host"+"\n port: "+port+"\n index: "+index+"\n path_to_ca_crt: "+path_to_ca_crt+"\n accessToken: "+accessToken);
String host=args[0];
String port=args[1];
String index=args[2];
String path_to_ca_crt=args[3];
String accessToken=args[4];
Path caCrtificatePath=Path.get(path_to_ca_crt);
CertificateFactory certFactory=
CertificateFactory.getInstance("X.509");
java.security.cert.Certificate trustedCa;
try (InputStream is=Files.newInputStream(caCrtificatePath)){
trustedCa=certFactory.generateCertificate(is);
}
KeyStore trustStore=keyStore.getInstance("pkcs12");
trustStore.load(null,null);
trustStore.setCertificateEntry("ca", trustedCa);
SSLContextBuilder sslContextBuilder=SSLContexts.custom()
.loadTrustMaterial(trustStore, null);
final javax.net.ssl.SSLContext sslContext=sslContextBuilder.build();
HostnameVerifier hostnameVerifier=NoopHostnameVerifier.INSTANCE;
SSLConnectionSocketFactory sslSocketFactory=
new SSLConnectionSocketFactory(sslContext, hostnameVerifier);
SchemeIOSessionStrategy httpsIOSessionStrategy=
new SSLIOSessionStrategy(sslContext, hostnameVerifier);
JestClientFactory factory=new JestClientFactory();
factory.setHttpClientConfig(new HttpClientConfig.Builder("https://"+host+":"+port)
.defaultSchemeForDiscoveredNodes("https")
.sslSocketFactory(sslSocketFactory)
.httpsIOSessionStrategy(httpsIOSessionStrategy)
.build()
);
JestClient client=factory.getObject();
String query="";
Search search=null;
Map<String,Object> header=new HashMap<>;
header.put("Authorization", "Basic "+accessToken);
search=new Search.Builder(query).addIndex.setHeader(header).build();
JestResult result=client.execute(search);
JsonObject jo=result.getJsonObject();
System.out.println("search result \n "+jo.toString());
}
}
Running project as Maven build and goal assempbly:single will produce JestToSecurES.jar with main class JestConnSecurES in its MANIFEST.MF file
Running/Testing java client on Linux
Notes:
-Place http.crt (mentioned above in this post) along with JestToSecurES.jar in particular directory
-Create/generate value of accessToken by encoding the value of auth_user:auth_user_password in Base64
-The index books has been initially created in the cluster and as previously noted the authenticated user has been granted permissions to monitor the cluster and all operations on indices
Allowed accessToken
java -jar JestToSecurES.jar [host_or_ip] [port] books http.crt [Base64(auth_user:auth_user_password)]
Console output will contain: ... "total":1,"successfull":1 ... "_index":"books","type":"book" ...
Forbidden accessToken (change previous value of [Base64(auth_user:auth_user_password)] somehow to corrupt it)
java -jar JestToSecurES.jar [host_or_ip] [port] books http.crt [Base64(auth_user:auth_user_password)]
Console output will contain: ... "error":{"root_cause"} ... "security_exception" ... "unable to authenticate user" ... for REST request [/books/_search]...
Goals:
-setup/configure fully secured elastic cluster with X-Pack (TLS/SSL for internal and external communication) on RHEL
-implement RBAC for authenticated user
-implement simple java client based on JEST API and test it with secured cluster
Summary of proceeded steps to achieve the goals
[1]Setup and configured 2 host (two nodes on each) ES-7.8.0 unsecured cluster on Linux
-Created index books
-Tested successfully unsecured cluster with curl
[2]Enabled x-pack on the cluster and configured TLS/SSL for internal communication between nodes (elasticsearch.yml)
-Implemented Role Based Access Control
--Created authenticated user, user password via elastic api commands (not using Kibana), created particular role
--Granted permissions to particular role and assigned the authenticated user for this role
-Tested successfully cluster with curl to make sure RBAC works as expected
[3]Configured TLS/SSL on HTTP to being able to make request with httpS
--changed elasticsearch.yml accordingly
--generated elastic-certificate.p12 with bin/elasticsearch-certutil
--extracted certificate from elastic-certificate.p12 into http.p12 with openssl
-Tested successfully fully secured cluster with curl
[4]Created simple java app based on JEST API
-Tested successfully fully secured elastic cluster with simple java client
To implement java client on JEST, reading elastic doc HLRC/LLRC allowed me to understand how to incorporate http.crt into java jest client.
I'll try to implement java client on HLRC/LLRC later on.
I agree with you, it is not easy to consult developers on this topic with different issues, because there are lots of dependencies like releases, used libs, system platforms, developers work on, different terminologies/meanings used for the same things or vice versa, etc., regardless elastic/x-pack doc is very detailed and contains lots of cross-references.
I think having simple use case sample for HLRC/LLRC java client with the steps I proceeded above from end-to-end, will allow developers to proceed with everything efficient and make the process more understandable and enjoyable.
Thank you all, elastic team for sharing your experience/knowledge globally and your help and support.
Next task is to make hadoop-elasticsearch/connector works with this TLS/SSL x-pack secured elastic cluster.
ES-Hive integration is not working on secured elastic cluster
The details are below
Elasticsearch 7.8.0 (basic license), Hive 1.2.1000.2.6.5.0-292
TLS/SSL secured cluster has been configured and tested successfully with curl command and java JEST client on RHEL
curl --cacert /path/to/http.crt -u estester:estester_pwd -XGET 'https://ip:port/_cat/nodes?v'
curl --cacert /path/to/http.crt -u estester:estester_pwd -XGET 'https://ip:port/_cat/indices/books?pretty'
elasticsearch.yml
# Security configuration
xpack.security.enabled: true
# TLS/SSL encryption inter-node communication
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: certs/elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: certs/elastic-certificates.p12
# TLS/SSL encryption http client communication
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.keystore.path: "http.p12"
xpack.security.http.ssl.truststore.path: "http.p12"
xpack.security.http.ssl.client_authentication: optional
xpack.security.transport.ssl.supported_protocols: [ "TLSv1.2", "TLSv1.1", "TLSv1"]
Add jars
hive> ADD JAR hdfs://xxxxx/tmp/elasticsearch-hadoop-7.8.0.jar;
hive> ADD JAR .../commons-httpclient-3.0.1.jar;
Create table
hive> create external table if not exists db_name.tbl_name (
id string,
title string,
year string)
STORED BY 'org.elasticsearch.hadoop.hive.EsStorageHandler'
TBLPROPERTIES(
"es.nodes"="xx.xxx.xx.xxx",
"es.port"="xxxx",
"es.resource.read"="books/book",
"es.read.operation"="index",
"es.index.read.missing.as.empty"="true",
"es.nodes.discovery"="true",
"es.read.metadata"="true",
"es.mapping.names"="id:_metadata._id, title:title, year:year",
"es.net.ssl"="true",
"es.net.ssl.cert.allow.self.signed"="true",
"es.net.ssl.keystore.type"="PKCS12",
"es.net.ssl.keystore.location"="hdfs:///xxxxx_domain/tmp/http.p12",
"es.net.ssl.truststore.location"="hdfs:///xxxxx_domain/tmp/http.p12",
"es.net.http.auth.user"="estester",
"es.net.http.auth.pass"="estester_pwd");
Table created successfully, select failed with the details below
hive> select * from db_name.tbl_name;
OK
Failed with exception java.io.IOException:org.elasticsearch.hadoop.EsHadoopIllegalArgumentException: Cannot detect ES version- this typically happens if network/elasticsearch cluster is not accessible or when targetting a WAN/Cloud instance without proper setting in 'es.nodes.wan.only'
Note: When setting and configuring elastic cluster, I generated the following certificate files:
elastic-certificates.p12, elastic-stack-ca.p12, http.p12, http.crt
These files reside also in hdfs:///xxxxx/tmp/
A few questions please:
Q1: which of the above mentioned certificate file should be used for es.net.ssl.keystore.location and es.net.ssl.truststore.location ?
Q2: what would be the correct value for es.net.ssl.keystore.type ?
Q3: is referencing directory like hdfs:///xxxxx_domain/tmp correct for es.net.ssl.keystore.location and es.net.ssl.truststore.location ?
Am I missing something in table's properties or elsewhere ?
Thanks in advance
Based on the context that I've gathered from reading this thread, I think you'll want to set the truststore to the http.p12 file. Since Hive is acting as a client connection to the cluster, you should be able to ignore the keystore setting, similar to how you configured JEST up above.
keystore type you have is correct: PKCS12
Unfortunately, we do not support pulling keystore/truststore files from HDFS. You can deploy your truststore in one of two ways:
- Add the file to the job's classpath, and refer to it by name in the
es.net.ssl.truststore.locationproperty. ES-Hadoop will search the classpath for the resource and open it via the classpath. Since it is part of the job's classpath, Hive will make sure it is on every node. - Arguably the easier choice if you have total control over your deployment, place the truststore on each node's local filesystem in your Hive cluster, at the same path location on each node. Refer to the file path using
file:///URL on thees.net.ssl.truststore.locationsetting. ES-Hadoop treats full file URI's as a local resource instead of one provided via the classpath.
A quick cursory check:
This isn't a valid property for ES-Hadoop. Read operations are always a scrolled search, this is not configurable.
I would avoid the dotted field name in the es.mapping.names property.
These settings can be provided in a keystore instead of in plaintext on the table properties if you would like: Security | Elasticsearch for Apache Hadoop [8.11] | Elastic
Hope that helps!
P.S. Just a quick clarification here: We've seen issues with Hive where adding a file to the Hive Classpath via the Hive settings is not sufficient to access it in ES-Hadoop. The way that we've found that works effectively is to use the ADD FILE HiveQL directive. I suggest just deploying the certificates to each node's local filesystem as a cleaner approach.
Thank you very much James
I'll follow up your guidelines and recommendations
The es-hadoop connector is not working yet
Here is what I did
Replaced these two entries in the above table properties
"es.net.ssl.keystore.location"="hdfs:///xxxxx_domain/tmp/http.p12",
"es.net.ssl.truststore.location"="hdfs:///xxxxx_domain/tmp/http.p12",
by one
"es.net.ssl.truststore.location"="file:///path_to_cert_on_local/http.p12",
Placed file http.p12 in the directory /path_to_cert_on_local/ on all 10 nodes/servers of Hadoop cluster
Started hive in debug mode
Added 2 jars: elasticsearch-hadoop-7.8.0.jar and commons-httpclient-3.0.1.jar
Added certificate file
ADD FILE /path_to_cert_on_local/http.p12
Executed create table and select from it
Log file shows
java.io.IOException:org.elasticsearch.hadoop.EsHadoopIllegalArgumentException: Cannot detect ES version- this typically happens if network/elasticsearch cluster is not accessible or when targetting a WAN/Cloud instance without proper setting in 'es.nodes.wan.only'
Caused by: org.elasticserch.hadoop.EsHadoopIllegalStateException: Cannot initialize SSL - Expected to find keystore file at [file:///path_to_cert_on_local/http.p12] but was unable to. Make sure that it is available on the classpath, or if not, that you have specified a valid URI.
The following entries in the table properties worked well without any issues when I tested es-hadoop connector with Non-secured es cluster
"es.read.operation"="index",
"es.mapping.names"="id:_metadata._id, title:title, year:year",
So, I'm still experiencing the issue.
Any recommendations will be appreciated.
I fixed the previous error by changing truststore location to
"es.net.ssl.truststore.location"="file:/path_to_cert_on_local/http.p12",`
Now when running sql select the LOG shows the following exception
... Open Connection to xx.xxx.xx.xxx:xxxx
...Loading keystore located at [file:/path_to_cert_on_local/http.p12]
...Content-Type: application/json
...Authorization: Basic LK...
...Usr-Agent: ...
...Host: xx.xxx.xx.xxx:xxxx
...Closing the connection httpclient.HttpConnection.java.closeSocketAndStreams - Exception when closing output
javax.net.ssl.SSLException: Connection has been shutdown: javax.net.ssl.SSLException: java.lang.RuntimeException Unexpected error:
java.security.InvalidAlgorithmParameterException:
the trustAnchors parameter must be non-empty
Anyone experienced and fixed this ?
It looks like your http.p12 is missing the trusted certificate entry that should have been generated by certutil.
What do you get from
keytool -list -keystore /path/to/http.p12 -storepass ""
(That's assuming your PKCS#12 file has no password, replace that final "" with the real password if you have one)
Thanks for the response Tim
When I generated the certificates with certutil for http.p12, I went with blank password option, simply pressed the "Enter" key at the prompt, as advised
The elasticsearch.yml (in this thread above) contains entries with certificates
# TLS/SSL encryption http client communication
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.keystore.path: "http.p12"
xpack.security.http.ssl.truststore.path: "http.p12"
and I tested the cluster already successfully with CURL and java jest client (in this thread above)
The output of
keytool -list -keystore /path/to/http.p12 -storepass ""
is as follows
Keystore type: PKCS12
Keystore provider: SUN
Your keystore contains 2 entries
ca, Nov 3, 2020, trustedCertEntry,
Certificate fingerprint (SHA1) 46:E3 .................... E1:19
http, Oct 7, 2020, PrivateKeyEntry,
Certificate fingerprint (SHA1) 1A:C4 .................... 45:2G
Any recommendation how to make it work ?
I could Not make it work with the PKCS12 type in table properties like this
"es.net.ssl.keystore.type"="PKCS12",
"es.net.ssl.truststore.location"="file:/path_to_cert_on_local/http.p12",
How to make it work in es-hadoop v.7.8.0 with es v.7.8.0 ?
I didn't find any successful solution on the forum.
I made the following configuration work successfully with the JKS type:
Imported the http.crt into truststore.jks with the command
keytool -import -trustcacerts -alias tls -file http.crt -keystore truststore.jks
and the truststore.jks has been generated
hive> create external table if not exists db_name.tbl_name (
id string,
title string,
year string)
STORED BY 'org.elasticsearch.hadoop.hive.EsStorageHandler'
TBLPROPERTIES(
"es.nodes"="xx.xxx.xx.xxx",
"es.port"="xxxx",
"es.resource.read"="books/book",
"es.read.operation"="index",
"es.index.read.missing.as.empty"="true",
"es.nodes.discovery"="true",
"es.read.metadata"="true",
"es.mapping.names"="id:_metadata._id, title:title, year:year",
"es.net.ssl"="true",
"es.net.ssl.cert.allow.self.signed"="true",
"es.net.ssl.keystore.type"="jks",
"es.net.ssl.truststore.location"="file:/xxxxx_domain/tmp/truststore.jks",
"es.net.http.auth.user"="estester",
"es.net.http.auth.pass"="estester_pwd");
This configuration works well, I was able to run select from table successfully, but I'd like to take off the user/password in plain text from table properties and incorporate them into keystore file and reference this file in table properties
Importing user/password settings into keystore file - Not working
To import "es.net.http.auth.user"="estester" and "es.net.http.auth.pass"="estester_pwd" settings into keystore file I followed up
To create keystore file I ran the command as guided in the documentation
java -classpath elasticsearch-hadoop-7.8.0.jar org.elasticsearch.hadoop.cli.keytool create
and esh.keystore file has been created; than I ran
java -classpath elasticsearch-hadoop-7.8.0.jar org.elasticsearch.hadoop.cli.keytool add es.net.http.auth.user
enter value for es.net.http.auth.user: [entered estester]
java -classpath elasticsearch-hadoop-7.8.0.jar org.elasticsearch.hadoop.cli.keytool add es.net.http.auth.pass
enter value for es.net.http.auth.pass: [entered estester_pwd]
and in table properties I used
"es.net.ssl.keystore.location"="file:/xxxxx_domain/tmp/esh.keystore"
instead of "es.net.http.auth.user"="estester" and "es.net.http.auth.pass"="estester_pwd"
I added file
hive> ADD FILE /path_to_file/esh.keystore;
Recreated table, did select from table and log shows the Exception
.....
java.io.IOException:org.elasticsearch.hadoop.EsHadoopIllegalArgumentException: Cannot detect ES version- this typically happens if network/elasticsearch cluster is not accessible or when targetting a WAN/Cloud instance without proper setting in 'es.nodes.wan.only'
.....
Caused by: org.elasticsearch.hadoop.EsHadoopIllegalState exception: Cannot initialize SSL - Get Key failed: AES SecretKeyFactory not available
.....
Caused by: java.security.UnrecoverableKeyException: Get Key failed: AES SecretKeyFactory not available
.....
Caused by: java.security.NoSuchAlgorithmException: AES SecretKeyFactory not available
Q:How to create keystore file properly and import user/password into it ?
Q2: Which value should be assigned to "es.net.ssl.keystore.pass" and where it comes from if it is needed at all ?
es-hadoop with esh.keystore is not working
I tried also the following configuration to move the entry es.net.http.auth.pass and value estester_pwd from table to esh.keystore file
Table
..........
"es.net.ssl"="true",
"es.net.ssl.cert.allow.self.signed"="true",
"es.net.ssl.keystore.type"="jks",
"es.net.ssl.truststore.location"="file:/path_to_truststore/truststore.jks",
"es.net.http.auth.user"="estester",
"es.net.ssl.truststore.location"="file:/path_to_keystore/esh.keystore");
Added entry es.net.http.auth.pass and its value estester_pwd to the esh.keystore
Select from table failed and gives the same exception as before
..........
Caused by: java.security.NoSuchAlgorithmException: AES SecretKeyFactory not available
..........
I tried to add the entry es.net.http.auth.pass and its value estester_pwd to esh.keystore with double, single quotes and without quotes at all - nothing is working.
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.