Testing TLS/SSL secured elastic cluster

Elastic Stack Elasticsearch
1

I configured 2 node es cluster with the TLS security section in es yml file using certificates in PKCS#12 format, native realm by default, basic license, elasticsearch-7.8.0-linux-86_64.tar.gz, RHEL 5.11.
Used command with no password:
bin/elasticsearch-certutil cert -out config/certs/elastic-certificates.p12 -pass ""
I placed generated file elastic-certificates.p12 on 2 nodes in config/certs/, so it is not host/node specific.
Elasticsearch.yml

.....
xpack.security.enabled: true
#TLS
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: certs/elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: certs/elastic-certificates.p12
#SSL
xpack.security.transport.ssl.enabled: true
xpack.security.http.ssl.keystore.path: certs/elastic-certificates.p12
xpack.security.http.ssl.truststore.path: certs/elastic-certificates.p12
xpack.security.http.ssl.client_authentication: optional
xpack.security.transport.ssl.supported_protocols: [ "TLSv1.2", "TLSv1.1", "TLSv1"]

Created custom role, assigned privileges to it; created custom user estester with estester_pwd, assigned the role to the user;

Testing TLS only use case - With the commented out the SSL section in config file (testing TLS only) I was able to successfully test the cluster with the curl command like
curl -u estester:estester_pwd -XGET "http://host:port/_cluster/health?pretty"

Testing TLS/SSL use case - After then I uncommented the SSL section in config file (to test SSL), restarted nodes
Ran

curl -u estester:estester_pwd -XGET "https://host:port/_cluster/health?pretty"

Es log:

... io.netty..handler.codec..DecoderException: javax.net.ssl.SSLException: Received fatal alert: uncnown.ca

Response on console:

SSL certificate problem, verify that the CA cert is ok. Details:
error : SSL routines: SSL3_GET_SERVER_CERTIFICATE:certificate verify failed
curl performs SSL certificate verification by default, using a "bundle" of Certificate Authority (CA) public keys (CA certs). The default bundle is named curl-ca-bundle.crt; you can specify an alternate file using the --cacert option. If this HTTPS server uses a certificate signed by a CA represented in the bundle, the certificate verification probably failed due to a problem with the certificate (it might be expired, or the name might not match the domain name in the URL). If you'd like to turn off curl's verification of the certificate, use the -k (or --insecure) option.

Any advise how this might be fixed ?
What curl command might be used to test SSL on the cluster in my use case?
Is there any command to convert passwordless elastic-certificates.p12 from PKCS#12 into PEM format ?

Thanks in advance

2

Hi, please format your codes and errors under preformatted text </> or backticks (```) as it is hard to read otherwise.

  1. When you start the service, did you see any errors in log?
  2. To test certs using curl, you can use
    curl --cacert /path/to/ca.crt -u elastic 'https://xxx:9200/_cat/nodes?v'
  3. To convert p12 to pem, you can use openssl for example see link below
    Make .p12 to .pem for kibana

Do share the outcome from 1 and 2.

3

Thank you for the response Kavier

1 - When I start nodes I can see in logs there were no errors; nodes started successfully, status of nodes are GREEN

2 - To test the cluster with the curl I need ca.crt which I don't have.
As I mentioned initially I generated passwordless (with no password provided) elastic-certificates.p12 as follows
bin/elasticsearch-certutil cert -out config/certs/elastic-certificates.p12 -pass ""
To convert it to PEM I ran command:
openssl pkcs12 -in certs/elastic-certificates.p12 -out certs/elastic-certificates.crt -nokeys
I has been prompted for password, clicked enter (because I have no password) and empty file elastic-certificates.crt has been created with zero size.

Q: how I can create ca.crt from my passwordless elastic-certificates.p12 ?

4

I'm not an expert for certs, but I wonder will it be different if you -out cert .pem instead?

openssl pkcs12 -in elastic-certificates.p12  -out elastic-ca.pem -nokeys

I always use below cheatsheet when dealing with certs issue.. hope it helps.

https://www.sslshopper.com/article-most-common-openssl-commands.html

5

Created .crt file:
openssl pkcs12 -in certs/elastic-certificates.p12 -out certs/elastic-certificates.crt -cacerts -nokeys
Ran request:
curl --cacert /path/to/elastic-certificates.crt -u elastic 'https://xx.xxx.xx.xxx:9200/_cat/nodes?v'
Response on console:
curl: (51) SSL: certificate subject name 'instance' does not match target host name 'xx.xxx.xx.xxx'
Elastic log: nothing changed after running the request

6

Can you run below to check on the cert?
openssl x509 -in certificate.crt -text -noout

7

Yes, ran the command successfully and can see the output on the console. I can't share it because I'm working on corp environment and chatting here from my personal laptop. The content of file looks good, it contains two sections -
Subject public key ... and X509v3 Basic constrains :critical CA:TRUE ...
The word "instance" is not in there

8

I think the response on console
curl: (51) SSL: certificate subject name 'instance' does not match target host name 'xx.xxx.xx.xxx'
means the client (curl) tries to match CN name 'instance' with the domain name in URL request. The question is why it takes the 'instance' and where the 'instance' come from ?

9

Looking back at the very first begining,

This cert is generated without CA

and this line is saying it cant find a CA

Is it possible for you to retry the SSL TLS setup following below document?

  1. Generate CA / CSR to sign by CA
  2. Generate node cert with above CA
  3. Move cert to ideal directory
  4. Put the cert config into elasticsearch.yml
10

Kavier, repeated the steps once again as you advised according the doc
In step 3 of es doc when generated elastic-certificates.crt, was prompted for host names, ips of hosts (I entered that but I don't want to use them)

Created .crt file as follows:
openssl pkcs12 -in certs/elastic-certificates.p12 -out certs/elastic-certificates.crt -cacerts -nokeys

Restarted nodes

Test/request the TLS/SSL secured cluster:
curl --cacert /path/to/elastic-certificates.crt -u elastic 'https://xx.xxx.xx.xxx:9200/_cat/nodes?v'

Response on console:
curl: (51) SSL: certificate subject name 'instance' does not match target host name 'xx.xxx.xx.xxx'
Elastic log: no new lines appeared in the log after running the request

Issue: wrong response Am I missing something (?)

Also, my elasticsearch.yml contains the lines
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.http.ssl.client_authentication: optional
and according to the doc the handshake process should not perform verification of matching the host/subject from the certificates against the host in URL, correct ?

When testing the cluster with curl, should I use the certificate file created by
openssl pkcs12 -in certs/elastic-certificates.p12 -out certs/elastic-certificates.crt -cacerts -nokeys
instead of certificate file created in step 3 of es doc ?

11

@warkolm Could you shed some light on this issue? Thanks!!

12

The original problem in your first post is that you have configured elasticsearch with a certificate that is signed by a custom CA so your operating system ( and curl by extension) doesn't know that it should trust it. You attempt to solve this by telling curl explicitly that it should trust this CA. ( Note that this should solve your problem for your machine and curl but not for anyone else )

Lets take things one by one.

  • Testing TLS only use case** .....
    Testing TLS/SSL use case**

    TLS and SSL are the same thing. TLS is a newer version of SSL and terms are used in literature interchangeably ( unfortunately ) .

  • xpack.security.transport.ssl.verification_mode: certificate: This is not a secure configuration. Any reason in specific that you want to disable hostname verification?

  • The latest problem you have is that you have created a certificate for a node and you tell it that the hostname to which it applies is X. Then, you try to access that node by Y . And your local tool ( curl ) tells you that the certificate is for X and not for Y and so, it can't validate it. You need to recreate the certificate and add the correct Subject Alternate Names ( with --dns and --ip )

  • was prompted for host names, ips of hosts (I entered that but I don't want to use them)

    If you add the IP addresses as SANs then you can remove the xpack.security.transport.ssl.verification_mode: certificate setting too and make your configuration much more secure.

HTH

1 Like
13

Thanks for your attention Ioannis, I understand your responses but still have some questions
I'm trying to setup secured es cluster as follows:

(a)setup/config RBAC for particular user/password
test with curl

(b)setup/config TLS/SSL secure inter-node communication with one single certificate which might be used for every node in the cluster
test with curl

(c)setup/config TLS/SSL for HTTP client, external communication - meaning java app/client will request the es secured cluster via https
test with curl

(d)test secured es cluster with java/app client via https
Also, no Kibana install/setup, all setup/config are done via es REST API

Currently I'm working on ES-7.8 poc cluster (RHEL 5.11) with two hosts and two nodes on each host (one is master, one is data), but real life cluster will contain up to 20 hosts and up to 20 nodes on each with the ability to add more hosts/nodes if needed.

(a)initially I setup RBAC on unsecured cluster for user:estester and pwd: estester_pwd with privileges "monitor" cluster and "all" for indices; tested this with curl successfully

(b)after then I setup/config TLS/SSL secure inter-node communication without hosts/ips/SANs in the certificate as described in steps 1 and 2 at es doc
Encrypting communications in Elasticsearchedit and generated certificate in step 2 without SAN in it.
I tested the cluster successfully with the "curl -k ..." and the elasticsearch.yml below
elasticsearch.yml
... cluster name, node name, etc, ...
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: certs/elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: certs/elastic-certificates.p12
xpack.security.transport.ssl.enabled: true
xpack.security.http.ssl.keystore.path: certs/elastic-certificates.p12
xpack.security.http.ssl.truststore.path: certs/elastic-certificates.p12
xpack.security.http.ssl.client_authentication: optional
xpack.security.transport.ssl.supported_protocols: [ "TLSv1.2", "TLSv1.1", "TLSv1"]

The reason for not using hostnames/IPs verification in the certificate for inter-node communication is the real life cluster will be running on hosts protected by corp firewall and will not be exposed for public external internet communication.
Another reason is to simplify the process of adding new hosts/nodes if needed and use the same certificate for inter-node communication

Q: is this setup/config without hosts/IPs in the certificate possible/reasonable taking into account having corp firewall on all hosts ?
Q: to my understanding this certificate can Not be used for external communication from app/java client to the cluster, correct ?

(c)then I generated certificate for external HTTP client communication, step 3 (optional) at es doc
When prompted for "Generate certificate per node? [y/N]" I selected the "y" and when prompted, provided node name and host name for the first host, first node, etc.
Unzipped generated zip file, placed http.p12 in the config dir of the first node.
Changed elasticsearch.yml per es doc as follows:
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: certs/elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: certs/elastic-certificates.p12
xpack.security.transport.ssl.enabled: true
xpack.security.http.ssl.keystore.path: "http.p12"
xpack.security.http.ssl.truststore.path: certs/elastic-certificates.p12
xpack.security.http.ssl.client_authentication: optional
xpack.security.transport.ssl.supported_protocols: ["TLSv1.2", "TLSv1.1", "TLSv1"]

Restarted nodes.
Converted http.p12 to http.crt
openssl pkcs12 -in http.p12 -out http.crt -cacerts -nokeys
Tested cluster successfully with expected response making request on the host, which are in the certificate, as follows
curl --cacert /path/to/http.crt -u estester:estester_pwd 'https://xx.xxx.xx.xxx:9200/_cat/nodes?v'

I still have some questions related to (c) above please
Q: when prompted with question "Generate certificate per node? [y/N]" and if I wish to generate single certificate that is valid across all the hostnames or ip addresses in the cluster, should I go with the "y" or "N" ?

Q: should I change the xpack.security.http.ssl.truststore.path: certs/elastic-certificates.p12 for xpack.security.http.ssl.truststore.path: "http.p12" in the above .yml

Q: with question "Generate certificate per node? [y/N]" (y or N) when prompted for node name, host name should I enter all node names (in my use case 4) and all host names (in my use case 2) OR just master nodes and hosts which will be used by java app client to request the cluster via https ?

Having all these final and correct setup/config in place, will allow to go with (d) testing es secured cluster with java/app client

Thanks a lot in advance

14

In this config (and earlier in this thread) you have

xpack.security.transport.ssl.enabled: true

twice, but do not have

xpack.security.http.ssl.enabled: true

If you want the node to be accessible over https, then you must enable http.ssl

If you want one single certificate for the whole cluster, then you do not want a certificate per node and should answer N to this question.

Yes, your certs/elastic-certificates.p12 file is intended to secure the transport protocol that the nodes use to connect to each other. For http you should use the http.p12 you generated.

You should enter every hostname/IP that any client (including curl, or administrative tools) will ever use to connect to your cluster. If in doubt, that should be every node.

just master nodes and hosts which will be used by java app client to request the cluster via https

It is not recommended to have clients connect directly to master nodes. If your cluster has dedicated master nodes, then they should be dedicated to being master nodes, and should not handle client requests.

15

Thanks for the quick response Tim.
That twice repeated line in configuration yml file is just typo error here, apology for the inconvenience. In the real cluster it was the correct line.
elasticsearch.yml

... cluster name, node name, etc, ...
# Security configuration
xpack.security.enabled: true
# TLS/SSL encryption inter-node communication
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: certs/elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: certs/elastic-certificates.p12
# TLS/SSL encryption http client communication
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.keystore.path: "http.p12"
xpack.security.http.ssl.truststore.path: "http.p12"
xpack.security.http.ssl.client_authentication: optional
xpack.security.transport.ssl.supported_protocols: [ "TLSv1.2", "TLSv1.1", "TLSv1"]

I re-generated certificates for inter-node and http client communication and was able to test different use cases with the curl and authorized user as follows

curl --cacert /path/to/http.crt -u estester:estester_pwd -XGET 'https://ip:port/_cat/nodes?v'
curl --cacert /path/to/http.crt -u estester:estester_pwd -XGET 'https://host:port/_cat/nodes?v'
curl -u estester:estester_pwd -XGET 'https://host:port/_cat/nodes?v'
curl --cacert /path/to/http.crt -XGET 'https://host:port/_cat/nodes?v'

etc.

Thank you all - Kavier, Ioannis, Albert Zaharovits and Tim Vernum - for your patience, help and generosity.

Now, when the x-pack secured cluster is configured and tested with the curl, I'm going to implement java app to test it with elastic cluster.

If you can reference me to simple java code samples to test secured elastic cluster with the client certificate and authorized user via https, it will be appreciated.

16

Running java client on Linux gives ClassNotFoundException
I implemented simple java client based on JEST API
Project contains jest-6.3.1.jar and x-pack-transport-7.8.1.jar
Compiled java code successfully, ran it on Linux and got the exception
Details are below

import java.io.InputStream;
import java.nio.file.Files;
import java.nio.file.Path;
import java.security.KeyStore;
import java.security.cert.Certificate;
import java.security.cert.CertificateFacrory;
import java.util.HashMap;
import java.util.Map;

import javax.net.ssl.HostnameVerifier;
import javax.net.ssl.SSLContext;

import org.apache.http.conn.ssl.NoopHostnameVerifier;
import org.apache.http.conn.ssl.SSLConnectionSocketFactory;
import org.apache.http.nio.conn.SchemeIOSessionStrategy;
import org.apache.http.nio.conn.SSLIOSessionStrategy;
import org.apache.http.ssl.SSLContextBuilder;
import org.apache.http.ssl.SSLContexts;

import com.google.gson.JsonObject;
import io.searchbox.client.JestClient;
import io.searchbox.client.JestClientFactory;
import io.searchbox.client.JestResult;
import io.searchbox.client.config.HttpClientConfig;
import io.searchbox.core.Search;

public class JestConnSecurES {
	public static void main() throws Exception {

String host=args[0];
String port=args[1];
String index=args[2];
String path_to_ca_crt=args[3];
String accessToken=args[4];

Path trustStorePath = Paths.get("/path/to/truststore.p12");
KeyStore truststore = KeyStore.getInstance("pkcs12");
try (InputStream is = Files.newInputStream(trustStorePath)) {
    truststore.load(is, keyStorePass.toCharArray());
}
SSLContextBuilder sslBuilder = SSLContexts.custom()
    .loadTrustMaterial(truststore, null);
final SSLContext sslContext = sslBuilder.build();

//CA certificate is available as PEM encoded file *.crt
Path caCrtificatePath=Path.get(path_to_ca_crt);
CertificateFactory certFactory=
		CertificateFactory.getInstance("X.509");
Certificate trustedCa;
try (InputStream is=Files.newInputStream(caCrtificatePath=Path)){
	trustedCa=certFactory.generateCertificate(is);
}

KeyStore trustStore=keyStore.getInstance("pkcs12");
trustStore.load(null,null);
trustStore.setCertificateEntry("ca", trustedCa);
SSLContextBuilder sslContextBuilder=SSLContexts.custom()
	.loadTrustMaterial(trustStore, null);
final SSLContext sslContext=sslContextBuilder.build();

HostnameVerifier hostnameVerifier=NoopHostnameVerifier.INSTANCE;

SSLConnectionSocketFactory sslSocketFactory=
	new SSLConnectionSocketFactory(sslContext, hostnameVerifier);
SchemeIOSessionStrategy httpsIOSessionStrategy=
	new SSLIOSessionStrategy(sslContext, hostnameVerifier);

JestClientFactory factory=new JestClientFactory();
factory.setHTTPClientConfig(new HttpClientConfig.Builder("https://"+host+":"+port)
	.defaultSchemeForDiscoveredNodes("https")
	.sslSocketFactory(sslSocketFactory)
	.httpsIOSessionStrategy(httpsIOSessionStrategy)
	.build()
);

JestClient client=factory.getObject();
String query="";
Search search=null;

Map<String,Object> header=new HashMap<>;
header.put("Authorization", "Basic "+accessToken);
search=new Search.Builder(query).addIndex.setHeader(header).build();

JestResultn result=client.execute(search);
JsonObject jo=result.getJsonObject();
System.out.println("search result \n "+jo.toString());
	}
}

Running java code on Linux
java -jar [java_main_class] [host] [port] [index] [path_to_ca_crt] [accessToken]

Console output

Error: A JNI error has occurred, please check your installation and try again
    Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/http/conn/socket/LayeredConnectionSocketFactory
    .......................................................................................................... 
    Caused by: java.lang.ClassNotFoundException: org.apache.http.conn.socket.LayeredConnectionSocketFactory

Am I missing some specific dependency, jar(s) ?
Any help will be appreciated

17

The code you shared has nothing Elasticsearch specific in it, so it's hard for us to troubleshoot what might be wrong.

Instead of treating ES as a generic REST API that you interact over HTTP , you can also use the High Level Rest Client, details are available in our docs: https://www.elastic.co/guide/en/elasticsearch/client/java-rest/current/java-rest-high.html

18

Thanks for the response Ioannis

In the shared code I tried to achieve:

[1]using JEST API, because there are lots of legacy code written on it for java client, currently communicating with ES-2.3/ES-5.6 clusters with ReadOnlyRest plugin and simplify the migration to newest ES-7.8
[2]use X-Pack security instead of ROR plugin for java client to communicate with secured ES-7.8

Q1: In your opinion, is it possible to use JEST API to communicate with X-Pack secured ES-7.8 cluster or the only High or Low Level Rest Clients should be used without mixing with JEST API ?

Q2: Which one is more preferable or must be used for java client implementation - High or Low Level Rest Client ?

Some explanation to shared code and issues:

public class JestConnSecurES {
	public static void main() throws Exception {

String host=args[0];
String port=args[1];
String index=args[2];
String path_to_ca_crt=args[3];
String accessToken=args[4];
//CA certificate is available as PEM encoded file *.crt
Path caCrtificatePath=Path.get(path_to_ca_crt);
CertificateFactory certFactory=CertificateFactory.getInstance("X.509");
Certificate trustedCa;
      try (InputStream is=Files.newInputStream(caCrtificatePath=Path)){
            trustedCa=certFactory.generateCertificate(is);
      }

KeyStore trustStore=keyStore.getInstance("pkcs12");
trustStore.load(null,null);
trustStore.setCertificateEntry("ca", trustedCa);
SSLContextBuilder sslContextBuilder=SSLContexts.custom()
      .loadTrustMaterial(trustStore, null);
final SSLContext sslContext=sslContextBuilder.build();

HostnameVerifier hostnameVerifier=NoopHostnameVerifier.INSTANCE;

SSLConnectionSocketFactory sslSocketFactory=
     new SSLConnectionSocketFactory(sslContext, hostnameVerifier);
SchemeIOSessionStrategy httpsIOSessionStrategy=
    new SSLIOSessionStrategy(sslContext, hostnameVerifier);

JestClientFactory factory=new JestClientFactory();
factory.setHTTPClientConfig(new 
    httpClientConfig.Builder("https://"+host+":"+port)
    .defaultSchemeForDiscoveredNodes("https")
    .sslSocketFactory(sslSocketFactory)
    .httpsIOSessionStrategy(httpsIOSessionStrategy)
    .build() );

JestClient client=factory.getObject();
String query="";
Search search=null;

Map<String,Object> header=new HashMap<>;
header.put("Authorization", "Basic "+accessToken);
search=new Search.Builder(query).addIndex.setHeader(header).build();

JestResultn result=client.execute(search);
JsonObject jo=result.getJsonObject();
System.out.println("search result \n "+jo.toString());
    }
}

[3]java client project in Eclipse/pom.xml references jest-6.3.1.jar and x-pack-transport-7.8.1.jar as advised at https://www.elastic.co/guide/en/elasticsearch/reference/current/setup-xpack-client.html
[4]first section of code from "Path trustStorePath = " till "final SSLContext sslContext=" I got as it is at https://www.elastic.co/guide/en/elasticsearch/client/java-rest/7.9/_encrypted_communication.html
because it allows to use the certificate file *.crt I tested with curl successfully on the cluster
In this section the "org.apache.http.conn.ssl.SSLContextBuilder" and "org.apache.http.conn.ssl.SSLContexts" say "SSLContextBuilder and SSLContexts deprecated"

Q3: Which packages/jars do contain these deprecated SSLContextBuilder and SSLContexts ?

[5]section of code from "JestClient client=factory.getObject()" till "JsonObject jo=result.getJsonObject()" has been tested successfully previously with ES-5.6/ROR plugin cluster
[6]shared java code constructs trustStorePath, sslBuilder, sslContext, caCrtificatePath, trustStore, sslContextBuilder, sslContext, sslSocketFactory, factory, client, header, search, result with requested index
The code accepts input arguments: host, port, index to search, path to certificate

Q4: based on [3]-[6] I'm not clear why you're saying "code you shared has nothing Elasticsearch specific in it" ?
I might be missing something, having not much experience in the field

I hope having answers to Q1-Q4 and references to practical examples can help me to proceed further in right direction with java client implementation.
Also will follow up your advise to look at High Level Rest Client

Many thanks in advance

19

Hi,

Apologies for the previous post, I didn't mean to put you off - sorry if it came out this way.

I have no working knowledge of this JEST API that you mention so I can't state an opinion. I know for a fact that {H,L}LRC work fine and that the transport client is deprecated in 7 and will be removed in 8 so my suggestion is to use the rest client.

I would use the HLRC unless you know why you have a specific reason to use the LLRC instead.

It's in the apache http client.

Look at your imports. You are not using anything from the elasticsearch transport client in your code, and there is nothing elasticsearch related in your code. You use some libraries to make a call to an HTTP REST API ( that, ok, happens to be elasticsearch in this case , but this is not relevant in that context )

In 3 you just added the xpack client as a dependency but your are not using it anywhere.

In 4 you are copying part of an example from a LLRC doc and attempt to apply it to a transport client setup, which won't work.

It's really hard for folks to provide generic consulting on development efforts in such kind of forums. I'd start by reading the rest client docs that were linked above. They have all the necessary setup and they contain examples and sections to get you going.

HTH

20

Thank you Ioannis for your attention and detailed answers, will read HLRC docs and give it a try.