Client did not trust this server's certificate, closing connection Netty4TcpChannel

Elastic Stack Elasticsearch
1

Hi Team,

I want to connect Eleasticsearch 6.8.0 via TransportClient vs 6.8.0, while connecting transport client I am getting below error on elasticsearch -
[2019-06-05T15:52:56,827][WARN ][o.e.x.c.s.t.n.SecurityNetty4Transport] [node-1] client did not trust this server's certificate, closing connection Ne
tty4TcpChannel{localAddress=0.0.0.0/0.0.0.0:9300, remoteAddress=/10.2.242.185:58362}

I want to use file/native security by basic license mode. Below are configuration used in elasticsearch.yml -
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: certs/certificate.p12
xpack.security.transport.ssl.truststore.path: certs/certificate.p12
xpack.license.self_generated.type: basic

Also below are properties added in JAVA client program -
TransportClient client = new PreBuiltXPackTransportClient(Settings.builder()
.put("cluster.name", "myClusterName")
.put("xpack.security.user", "test_user:x-pack-test-password")
.put("xpack.ssl.certificate_authorities", "/path/to/ca.crt")
.put("xpack.security.transport.ssl.enabled", "true")

I am using company/organization provided P12 trustore/keystore.

I am able to successfully connect Logstash and Kibana with basic userid and password. But not able to connect our client program to elasticsearch.

Basically we want to connect our client transport program like-wise kibana and logstash are connected with different urserid and role.
We are not looking for SSL connectivity between transport client, we want to connect it via HTTP communication with userid and password.

Please let us know, if we have missed any configuration.

2 Likes
2

You can't enable security without also enabling TLS for the transport layer. Also, while username and password is used for authenticating your client to Elasticsearch, but if you don't enable TLS for this communication channel, all data (including the passwords) are sent in plaintext from your client to elasticsearch and can be sniffed.

Is there any specific reason that you don't want to use TLS for client communication ?

Transport client doesn't communicate over http either way.
Is there any specific reason for using the transport client instead of the High Level Rest Client ? The transport client is deprecated in 7.0.0 and will be removed in 8.0.0.

That said, the errors you get above are because you have enabled TLS for the transport layer and you have tried to enable TLS in the transport client also, although unsuccessfully. I can see you have copied the configuration from Java Client and security | Elasticsearch Guide [7.1] | Elastic but you haven't shared with us how you have created the ca.crt Once you tell us a little more about what you want to achieve, we ca guide you better. For now, there are too many alternatives to know what to suggest solutions for.

3

Our existing application is based on TranportClient, hence we want to go ahead with TransportClient instead of High Level Rest Client at this moment. we have a plan to migrate it to High Level Rest Client.

From 6.8.0, we want to leverage basic authentication feature with existing infrastructure setup. As such we don't have any issue to switch to TLS based client communication. But if it is possible that transport client can communicate to elasticsearch without TLS, then by simply passing user id and password, how it will communicate it with elasticsearch?

We are currently using our organization specific CA certificate and P12 trustore/keystore.

Yes, your are right. I have tried to follow steps which are mentioned in https://www.elastic.co/guide/en/elasticsearch/reference/6.8/configuring-tls.html
for enabling security feature at elasticsearch.
And for transport client, I have followed steps which are mentioned in https://www.elastic.co/guide/en/elastic-stack-overview/6.8/java-clients.html

Looks like we have enabled TLS at both places.

Is it possible without enabling TLS feature I can use basic security feature in transport client as well?

We don't have any issues while communicating with Logstash and Kibana, but we are struggling with transport client communication.

Also I have tried to enable TLS at elasticsearch, but still I am getting same error which was mentioned in the beginning.

Also if I remove below line from JAVA client program

.put("xpack.security.transport.ssl.enabled", "true")

, then getting below error -
[2019-06-06T14:41:25,741][WARN ][o.e.t.TcpTransport ] [node-1] exception caught on transport layer [Netty4TcpChannel{localAddress=0.0.0.0/0.0.0.
0:9300, remoteAddress=/10.2.242.185:53780}], closing connection
io.netty.handler.codec.DecoderException: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 455300000069000000000000000208004d3603010d
417574686f72697a6174696f6e2e42617369632064484a68626e4e7762334a3058324e736157567564463931633256794f6b316b625541784d6a4d3d0016696e7465726e616c3a7463702f
68616e647368616b650004e38cf302

4

I'm not sure I understand your question, can you please rephrase it ?

I'd strongly suggest against this for reasons shared above.

You have done most of the configuration already either way, the only problem with your setup now feels to be that this "/path/to/ca.crt" is wrong. This needs to contain the CA certificate, that is also the one that signed the certificates that are used in the certificate.p12 you have in ES. Did you get both certificate.p12 and ca.crt from somwhere in your company, or did you export the ca.crt from the certificate.p12 yourself ? If so, how did you do this ?

This is expected behavior. Elasticsearch is expecting a connection over TLS, your client attempts to communicate without TLS, and the connection fails.

5

certificate.p12 and ca.crt both files, I have received from company.

6

It looks like that the certificate that is contained in certificate.p12 (which is the one used by Elasticsearch for TLS on the transport layer) is not signed by the CA certificate that is contained in the ca.crt file. Can you speak to someone at your company and get the correct CA certificate ?

Logstash and Kibana both communicate with elasticsearch on the HTTP layer and not the transport layer.

7

The ca.crt and certificate.p12 files were received from our certificate registry portal where everyone is getting from that portal only. We are using these certificates in across the company for different application. I don't think, there is an issue with ca.crt and certificate.p12 file.

Meanwhile I have did some testing around this -

  1. firstly I disabled verficate mode in elasticsearch.yml file -
    xpack.security.transport.ssl.verification_mode: none

getting same error - client did not trust this server's certificate, closing connection

  1. Then I tried with removing below two properties from transport client program -
    .put("xpack.security.user", "transport_client_user:Pass@123")
    .put("xpack.ssl.certificate_authorities", "ca.crt")

even I am getting same error - client did not trust this server's certificate, closing connection

Only I am getting different error, whenever I am removing -
.put("xpack.security.transport.ssl.enabled", "true")

getting error as -not an SSL/TLS record

Looks like something wired configuration or settings. please guide here how to proceed further.

8

Your problem is that the client doesn't trust the certificate of Elasticsearch. What you configured here is Elasticsearch to not validate the certificate of the client, this shouldn't have any effect to the problem you have and it doesn't, as you observed.

Your problem is that the client doesn't trust the certificate of Elasticsearch.

What you tried here is to
a) have your client make requests to Elasticsearch without a username and password , which is unrelated to your problem
b) Remove the configuration setting that would allow your client to trust the Elasticsearch certificate, which - as expected - didn't change anything.

Your problem is that the client doesn't trust the certificate of Elasticsearch.

What you tried to do here is to have your client not connect via SSL to Elasticsearh, which as expected, did not work.
I already explained before why you get this. Elasticsearch is expecting TLS connections on the transport layer and you go and configure your client to not use TLS, so Elasticsearch fails with a clear message - as expected.

Can't you at least verify this is the truth, instead of dismissing the idea ?

Export the elasticsearch certificate from the certificate.p12:

openssl pkcs12 -in certificate.p12 -clcerts -nokeys > elasticsearch.crt

Try to validate that ca.crt is the CA that has signed the elasticsearch.crt and share the output :

openssl verity -CAfile ca.crt elasticsearch.crt
9

Thanks for the detailed description, and correcting the approaches on reaching out exact root cause.

By execution of above openssl commands, I am getting below error on root and group root CA certificates -

  1. Root CA - error 2 at 1 depth lookup:unable to get issuer certificate
  2. Group Root CA - error 20 at 0 depth lookup:unable to get local issuer certificate
10

It is more helpful when you post the command and the whole output instead.

The result reinforces my assumption that the ca.crt is not the correct one as the certificate in certificate.p12 cannot be verified by it. Please talk to your company and get the correct certificates, we cant help you any more until you do that.

11

Now I have created CA file from below openssl command -
openssl pkcs12 -in certificate.p12 -cacerts -nokeys -chain | sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' > cacerts.cer

And I have verified CA file as per command you have given -

C:\Users\Downloads\elasticsearch-6.8.0\config\certs>openssl
OpenSSL> verify -CAfile cacerts.cer elasticsearch.crt
elasticsearch.crt: OK
OpenSSL>

Then I have used same cacerts.cer file in transport client program -
.put("xpack.ssl.certificate_authorities", "C:/Users/Downloads/keystore/cacerts.cer")
but getting same error as
"client did not trust this server's certificate, closing connection"

12

I can only assume that hostname verification fails ( i.e. your certificate is issued for a different domain name/IP Address than the one you attempt to connect to )

  1. Check what you use to connect to Elasticsearch from the client ( find the .addTransportAddress(XXX) in your PreBuiltXPackTransportClient ) and take a note of that IP Address.

  2. What's the output of

    openssl s_client -connect XXXX:9300 -CAfile C:/Users/Downloads/keystore/cacerts.cer

    ?

If the above fails also, you can either:

  • Get a proper certificate that has the IP Addresses of your Elasticserach nodes as Subject Alternate Names
  • Disable hostname verification (at your own risk) by setting
     .put("xpack.ssl.verification_mode", "certificate")
1 Like
13

Below is the output by executing command of #2 -
C:\Users\rupnsur>openssl s_client -connect 10.2.242.185:9300 -CAfile C:/Users/Downloads/keystore/cacerts.cer
Loading 'screen' into random state - done
CONNECTED(00000168)
17908:error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol:.\ssl\s23_clnt.c:610:

And at elasticsearch log, we have received below error message -
[2019-06-07T14:46:48,433][WARN ][o.e.t.TcpTransport ] [node-1] exception caught on transport layer [Netty4TcpChannel{localAddress=0.0.0.0/0.0.0.
0:9300, remoteAddress=/10.2.242.185:61024}], closing connection
io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: SSLv2Hello is disabled

Meanwhile i'll work on suggestions which you have mentioned in the bottom and share the result.

14

This has probably nothing to do with what we're dealing so far, it's just that openssl on your machine uses sslv2 by default ( which is strange but still, unrelated to previous errors )

You can specify the protocol version by setting one of -tls1_1, -tls1_2 in your openssl s_client command.

15

Now I have received new certificate.p12 and CA.cer file for IP Addresses.

Also I have followed CA certificate Openssl verify, I got 'OK'

Afterwards I tried to connect to elasticsearch -
C:\Users\rupnsur>openssl s_client -connect 10.2.242.185:9300 -CAfile C:/Users/Downloads/keystore/cacerts.cer

but I have received different error now -
[2019-06-07T19:58:45,128][WARN ][o.e.t.TcpTransport ] [node-1] exception caught on transport layer [Netty4TcpChannel{localAddress=0.0.0.0/0.0.0.
0:9300, remoteAddress=/10.2.242.185:63808}], closing connection
io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: null cert chain

Afterwards I have tried to put below configuration in elasticsearch.yml -
xpack.security.transport.ssl.client_authentication: none

And I am able to successfully connect to elasticsearch. No exceptions found.
I don't know exactly, what will be impact if I use above configuration? pls explain.

16

I can't see how this plays any role. You didn't have this set in the config you have shared with us previously and the default value for this is none, so adding the setting explicitly shouldn't make any difference at all.

This setting controls whether the client is also required to present a certificate and key for client TLS authentication. This should not be relevant here. If your client can connect with the current settings, then you're good to go.

This was badly misspoken, see @TimV post below.

17

Thanks for all the help and detailed description.

1 Like
18

This is a bad idea.
It means that any client can connect to your elasticsearch cluster (using transport protocol) without needing a client certificate. This opens up your cluster to having a rogue client connect and send malicious commands.

That is true for the http SSL context. For transport it defaults to required.

19

We are passing total 4 parameter -

  1. xpack.security.user
  2. xpack.ssl.verification_mode
  3. xpack.security.transport.ssl.enabled
  4. xpack.ssl.certificate_authorities

Whenever I am using correct CA certificate, then I am getting below error -
[2019-06-07T19:58:45,128][WARN ][o.e.t.TcpTransport ] [node-1] exception caught on transport layer [Netty4TcpChannel{localAddress=0.0.0.0/0.0.0.
0:9300, remoteAddress=/10.2.242.185:63808}], closing connection
io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: null cert chain

So you mean "xpack.security.user" credentials are not playing any role in terms of authentication and certificate based communication plays an important role in protecting cluster data.

Can you guide me on resolving this issue so that I can achieve basic level of security.

20

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.