Testing TLS/SSL secured elastic cluster

np, hope this helps you further!

If you have specific questions, issues while trying to use the rest client, folks in the forums will do their best to help out.

I implemented and successfully tested java JEST client as follows

pom.xml

<dependency>
   <groupId>io.searchbox</groupId>
   <artifactId>jest</artifactId>
   <version>6.3.1</version>
</dependency>
<dependency>
   <groupId>io.searchbox</groupId>
   <artifactId>jest-common</artifactId>
   <version>6.3.1</version>
</dependency>

Source code of project JestToSecurES in Eclipse (Neon)

import java.io.InputStream;
import java.nio.file.Files;
import java.nio.file.Path;
import java.security.KeyStore;
import java.security.cert.CertificateFacrory;
import java.util.HashMap;
import java.util.Map;

import javax.net.ssl.HostnameVerifier;

import org.apache.http.conn.ssl.NoopHostnameVerifier;
import org.apache.http.conn.ssl.SSLConnectionSocketFactory;
import org.apache.http.nio.conn.SchemeIOSessionStrategy;
import org.apache.http.nio.conn.SSLIOSessionStrategy;
import org.apache.http.ssl.SSLContextBuilder;
import org.apache.http.ssl.SSLContexts;

import com.google.gson.JsonObject;

import io.searchbox.client.JestClient;
import io.searchbox.client.JestClientFactory;
import io.searchbox.client.JestResult;
import io.searchbox.client.config.HttpClientConfig;
import io.searchbox.core.Search;

public class JestConnSecurES {
	public static void main() throws Exception {

System.out.println("Running main class JestConnSecurES of project JestToSecurES);	
System.out.println("Input arguments: "+"\n host: "+host"+"\n port: "+port+"\n index: "+index+"\n path_to_ca_crt: "+path_to_ca_crt+"\n accessToken: "+accessToken);
	
String host=args[0];
String port=args[1];
String index=args[2];
String path_to_ca_crt=args[3];
String accessToken=args[4];

Path caCrtificatePath=Path.get(path_to_ca_crt);
CertificateFactory certFactory=
	CertificateFactory.getInstance("X.509");
java.security.cert.Certificate trustedCa;
try (InputStream is=Files.newInputStream(caCrtificatePath)){
	trustedCa=certFactory.generateCertificate(is);
}

KeyStore trustStore=keyStore.getInstance("pkcs12");
trustStore.load(null,null);
trustStore.setCertificateEntry("ca", trustedCa);
SSLContextBuilder sslContextBuilder=SSLContexts.custom()
	.loadTrustMaterial(trustStore, null);
final javax.net.ssl.SSLContext sslContext=sslContextBuilder.build();
	
HostnameVerifier hostnameVerifier=NoopHostnameVerifier.INSTANCE;

SSLConnectionSocketFactory sslSocketFactory=
	new SSLConnectionSocketFactory(sslContext, hostnameVerifier);
SchemeIOSessionStrategy httpsIOSessionStrategy=
	new SSLIOSessionStrategy(sslContext, hostnameVerifier);

JestClientFactory factory=new JestClientFactory();
factory.setHttpClientConfig(new HttpClientConfig.Builder("https://"+host+":"+port)
	.defaultSchemeForDiscoveredNodes("https")
	.sslSocketFactory(sslSocketFactory)
	.httpsIOSessionStrategy(httpsIOSessionStrategy)
	.build()
);

JestClient client=factory.getObject();
String query="";
Search search=null;

Map<String,Object> header=new HashMap<>;
header.put("Authorization", "Basic "+accessToken);
search=new Search.Builder(query).addIndex.setHeader(header).build();

JestResult result=client.execute(search);
JsonObject jo=result.getJsonObject();
System.out.println("search result \n "+jo.toString());
	}
}

Running project as Maven build and goal assempbly:single will produce JestToSecurES.jar with main class JestConnSecurES in its MANIFEST.MF file

Running/Testing java client on Linux

Notes:
-Place http.crt (mentioned above in this post) along with JestToSecurES.jar in particular directory
-Create/generate value of accessToken by encoding the value of auth_user:auth_user_password in Base64
-The index books has been initially created in the cluster and as previously noted the authenticated user has been granted permissions to monitor the cluster and all operations on indices

Allowed accessToken
java -jar JestToSecurES.jar [host_or_ip] [port] books http.crt [Base64(auth_user:auth_user_password)]
Console output will contain: ... "total":1,"successfull":1 ... "_index":"books","type":"book" ...

Forbidden accessToken (change previous value of [Base64(auth_user:auth_user_password)] somehow to corrupt it)
java -jar JestToSecurES.jar [host_or_ip] [port] books http.crt [Base64(auth_user:auth_user_password)]
Console output will contain: ... "error":{"root_cause"} ... "security_exception" ... "unable to authenticate user" ... for REST request [/books/_search]...

Goals:
-setup/configure fully secured elastic cluster with X-Pack (TLS/SSL for internal and external communication) on RHEL
-implement RBAC for authenticated user
-implement simple java client based on JEST API and test it with secured cluster

Summary of proceeded steps to achieve the goals
[1]Setup and configured 2 host (two nodes on each) ES-7.8.0 unsecured cluster on Linux
-Created index books
-Tested successfully unsecured cluster with curl
[2]Enabled x-pack on the cluster and configured TLS/SSL for internal communication between nodes (elasticsearch.yml)
-Implemented Role Based Access Control
--Created authenticated user, user password via elastic api commands (not using Kibana), created particular role
--Granted permissions to particular role and assigned the authenticated user for this role
-Tested successfully cluster with curl to make sure RBAC works as expected
[3]Configured TLS/SSL on HTTP to being able to make request with httpS
--changed elasticsearch.yml accordingly
--generated elastic-certificate.p12 with bin/elasticsearch-certutil
--extracted certificate from elastic-certificate.p12 into http.p12 with openssl
-Tested successfully fully secured cluster with curl
[4]Created simple java app based on JEST API
-Tested successfully fully secured elastic cluster with simple java client

To implement java client on JEST, reading elastic doc HLRC/LLRC allowed me to understand how to incorporate http.crt into java jest client.
I'll try to implement java client on HLRC/LLRC later on.

I agree with you, it is not easy to consult developers on this topic with different issues, because there are lots of dependencies like releases, used libs, system platforms, developers work on, different terminologies/meanings used for the same things or vice versa, etc., regardless elastic/x-pack doc is very detailed and contains lots of cross-references.

I think having simple use case sample for HLRC/LLRC java client with the steps I proceeded above from end-to-end, will allow developers to proceed with everything efficient and make the process more understandable and enjoyable.

Thank you all, elastic team for sharing your experience/knowledge globally and your help and support.

Next task is to make hadoop-elasticsearch/connector works with this TLS/SSL x-pack secured elastic cluster.

ES-Hive integration is not working on secured elastic cluster

The details are below

Elasticsearch 7.8.0 (basic license), Hive 1.2.1000.2.6.5.0-292
TLS/SSL secured cluster has been configured and tested successfully with curl command and java JEST client on RHEL

curl --cacert /path/to/http.crt -u estester:estester_pwd -XGET 'https://ip:port/_cat/nodes?v'
curl --cacert /path/to/http.crt -u estester:estester_pwd -XGET 'https://ip:port/_cat/indices/books?pretty'

elasticsearch.yml

# Security configuration
xpack.security.enabled: true
# TLS/SSL encryption inter-node communication
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: certs/elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: certs/elastic-certificates.p12
# TLS/SSL encryption http client communication
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.keystore.path: "http.p12"
xpack.security.http.ssl.truststore.path: "http.p12"
xpack.security.http.ssl.client_authentication: optional
xpack.security.transport.ssl.supported_protocols: [ "TLSv1.2", "TLSv1.1", "TLSv1"]

Add jars

hive> ADD JAR hdfs://xxxxx/tmp/elasticsearch-hadoop-7.8.0.jar;
hive> ADD JAR .../commons-httpclient-3.0.1.jar;

Create table

hive> create external table if not exists db_name.tbl_name (
id string,
title string,
year string)
STORED BY 'org.elasticsearch.hadoop.hive.EsStorageHandler'
TBLPROPERTIES(
"es.nodes"="xx.xxx.xx.xxx",
"es.port"="xxxx",
"es.resource.read"="books/book",
"es.read.operation"="index",
"es.index.read.missing.as.empty"="true",
"es.nodes.discovery"="true",
"es.read.metadata"="true",
"es.mapping.names"="id:_metadata._id, title:title, year:year",
"es.net.ssl"="true",
"es.net.ssl.cert.allow.self.signed"="true",
"es.net.ssl.keystore.type"="PKCS12",
"es.net.ssl.keystore.location"="hdfs:///xxxxx_domain/tmp/http.p12",
"es.net.ssl.truststore.location"="hdfs:///xxxxx_domain/tmp/http.p12",
"es.net.http.auth.user"="estester",
"es.net.http.auth.pass"="estester_pwd");

Table created successfully, select failed with the details below

hive> select * from db_name.tbl_name;
OK
Failed with exception java.io.IOException:org.elasticsearch.hadoop.EsHadoopIllegalArgumentException: Cannot detect ES version- this typically happens if network/elasticsearch cluster is not accessible or when targetting a WAN/Cloud instance without proper setting in 'es.nodes.wan.only'

Note: When setting and configuring elastic cluster, I generated the following certificate files:
elastic-certificates.p12, elastic-stack-ca.p12, http.p12, http.crt
These files reside also in hdfs:///xxxxx/tmp/

A few questions please:
Q1: which of the above mentioned certificate file should be used for es.net.ssl.keystore.location and es.net.ssl.truststore.location ?

Q2: what would be the correct value for es.net.ssl.keystore.type ?

Q3: is referencing directory like hdfs:///xxxxx_domain/tmp correct for es.net.ssl.keystore.location and es.net.ssl.truststore.location ?

Am I missing something in table's properties or elsewhere ?

Thanks in advance