Exception caught on transport layer (es6.4.2)

Elastic Stack Elasticsearch
1

Hi !
I'm trying to reach a cluster (ES6.4.2) with SSl enabled.
I'm using the java low-level API and the TransportClient.

I can read/write on the cluster when SSL is disabled, but not when it is on.
I don't know which parameter I should use in my TransportClient ?

Currenty I have the following :

.put("xpack.security.transport.ssl.enabled", "true")
.put("xpack.security.transport.ssl.truststore.path", xxx)
.put("xpack.security.transport.ssl.truststore.password", xxx)

Here is the elasticsearch.yml :

discovery.zen.minimum_master_nodes: 1
cluster.name: elastic_cluster6
node.name: node-6.4.2
node.master: true
node.data: true
transport.host: 0.0.0.0
transport.tcp.port: 9302
http.port: 9202
network.host: 0.0.0.0
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: certs/elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: certs/elastic-certificates.p12
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.keystore.path: certs/elastic-certificates.p12
xpack.security.http.ssl.truststore.path: certs/elastic-certificates.p12

And of course, here is the error :

[2019-01-11T16:21:23,035][WARN ][o.e.x.s.t.n.SecurityNetty4ServerTransport] [node-6.4.2] exception caught on transport layer [NettyTcpChannel{localAddress=0.0.0.0/0.0.0.0:9302, remoteAddress=/192.168.150.3:37466}], closing connection
io.netty.handler.codec.DecoderException: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 455300000027000000000000000208004d3603000016696e7465726e616c3a7463702f68616e647368616b6500
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:459) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:545) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:499) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.16.Final.jar:4.1.16.Final]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_191]
Caused by: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 455300000027000000000000000208004d3603000016696e7465726e616c3a7463702f68616e647368616b6500
        at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1106) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1162) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]
        ... 15 more
2

Hi @aparent,

This is covered in our documentation , please take a look there and we'll be happy to help if you encounter any additional issues

3

Yes I'm aware of your documentation, but I don't know what I did wrong :confused:
Could you please give me a hint ?

4

It's rather difficult to guess that with the current information. The settings look fine, assuming you actually pass them to your TransportClient initialization. Do you mind sharing a larger part of your TransportClient related code ?

5

Here is the code used to create the client :

public org.elasticsearch.client.Client create() throws Exception {
			org.elasticsearch.common.settings.Settings settings = org.elasticsearch.common.settings.Settings
					.builder().put("cluster.name", "elastic_cluster6")
					.put("xpack.security.http.ssl.enabled", "true")
					.put("xpack.security.http.ssl.truststore.path", "/home/keystore/es6.p12")
					.put("xpack.security.http.ssl.truststore.password", xxx) 
					.put("xpack.security.transport.ssl.enabled", "true") 
					.put("xpack.security.transport.ssl.truststore.path", "/home/keystore/es6.p12") //
					.put("xpack.security.transport.ssl.truststore.password", xxx)

					.put("xpack.ssl.verification_mode", "certificate") //

					.build();

			String transport = "xxx";

			org.elasticsearch.client.transport.TransportClient transportClient = new org.elasticsearch.xpack.client.PreBuiltXPackTransportClient(
					settings);


			transportClient.addTransportAddress(new org.elasticsearch.common.transport.TransportAddress(xxx));


			return transportClient;
		}

It works weel when SSL/TLS is off, so I guess the errors I got are configuration related.

I tried using the ES/Hadoop API and it works with SSL/TLS enabled (I used the ES_NET_SSL_CERT_ALLOW_SELF_SIGNED option).
Unfortunately, I can't rely on this API due to several constraint.

If you need more information, please ask me !
Thanks for the help ! :slight_smile:

6

Please read through the documentation again. See, in specific, steps 1,2 and 4 from the "Configuring the Transport Client to work with a Secured Cluster" section. Your settings are missing the client credentials and the client’s key and certificate

7

When I use the ES/Hadoop API I don't specify the client's key and certificate, yet, it works with SSL/TLS enabled.
I don't understand why I would add parameters in the transport client if they are optional in another API ?
Please, correct me if I'm wrong so I can understand my mistake !

Again, thank you !

8

That shouldn't be the case. Elasticsearch for hadoop uses the transport layer to communicate with elasticsearch and if you have TLS enabled in the elasticsearch configuration for the transport layer, then you need to configure the key information in ES Hadoop. Can you share the configuration that works without key material in ES Hadoop and share some information on how you verified that it works ?

If for some reason, you want to disable TLS client authentication in the transport layer, the instructions are also linked from the documentation that I have shared above.

9

Here are the settings for ES/Hadoop API :

java.util.Map<String, String> config = new java.util.HashMap<String, String>();

config.put(org.elasticsearch.hadoop.cfg.ConfigurationOptions.ES_NET_USE_SSL, "true");
config.put(org.elasticsearch.hadoop.cfg.ConfigurationOptions.ES_NET_SSL_CERT_ALLOW_SELF_SIGNED, "true");
String es_truststorepath = "path_to_the_truststore";
config.put(org.elasticsearch.hadoop.cfg.ConfigurationOptions.ES_NET_SSL_TRUST_STORE_LOCATION, es_truststorepath);
config.put(org.elasticsearch.hadoop.cfg.ConfigurationOptions.ES_NET_SSL_TRUST_STORE_PASS, "password");

[...]

org.apache.spark.api.java.JavaPairRDD<String, String> esRdd = org.elasticsearch.spark.rdd.api.java.JavaEsSpark.esJsonRDD(ctx, config);

I used the configuration from my first message, and to verify that it works, I wrote and read some data.
The "ElasticSearch webpage" only loads with https://xxx.xxx.xxx.xxx:9202
When I disable SSL/TLS I can access it through http://xxx.xxx.xxx.xxx:9202

10

I misspoke in my previous post when I said

Elasticsearch for hadoop uses the transport layer to communicate with elasticsearch

. Elasticsearch for hadoop uses the http layer to communicate to Elasticsearch, not the transport one. In the http layer, TLS client authentication is not enabled by default and this is why you can use the the ES/Hadoop API with the specified configuration as you showed above.

The Elasticsearch transport client, as the name implies, uses the transport layer where TLS client authetnication is enabled by default, so you need to provide a key and certificate for your client in order to connect to an Elasticsearch node.

Since the transport client and the ES/Hadoop API use different layers to communicate with Elasticsearch, there are bound to be differences in the behavior and default settings between the two.

This is you accessing Elasticsearch via your browser on the http layer, and thus is unrelated to the transport client.

If you want to use the transport client, you either need to provide a key and certificate for TLS client authentication as the documentation shows, or you need to disable the client authentication. Alternatively, and this is probably a good idea nevertheless, you can use the High Level Rest Client as the Transport Client will be deprecated in 7.0 and removed in 8.0

2 Likes
11

Thanks a lot for your help, I'll keep this topic open for a few days.
As soon as I've fixed my code I'll close it ! :slight_smile:

12

I have a new question regarding your documentation :

The output is a single PKCS#12 keystore that includes the node certificate, node key, and CA certificate.

Isn't the generated keystore sufficient to use SSL/TLS over the transport layer ?

Plus, i don't know what to think about the error I got :

[o.e.x.s.t.n.SecurityNetty4ServerTransport] [node-6.4.2] exception caught on transport layer [NettyTcpChannel{localAddress=0.0.0.0/0.0.0.0:9302, remoteAddress=/xxx.xxx.xxx.xxx:44058}], closing connection

and below

Caused by: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 455300000027000000000000000108004d3603000016696e7465726e616c3a7463702f68616e647368616b6500

If I decode this 455300000027000000000000000108004d3603000016696e7465726e616c3a7463702f68616e647368616b6500

I have : ES'M6internal:tcp/handshake but what does it mean ? :thinking:

13

This is the first message sent when an Elasticsearch node connects to another Elasticsearch node, and means that the sender believes this to be a plaintext connection but the recipient believes it should be secured.

14

@aparent as I mentioned in the GH issue where you first asked this question: You get this error because your transport client attempts to connect via the transport layer to you Elasticsearch node without TLS but your Elasticsearch node is configured to only accept connections over TLS. This is why we are trying to assist you in this thread in configuring your Transport client to use TLS.

I am not sure I see what the question is in the statement above.. What in the wording of our documentation contradicts the statement that the generated keystore is sufficient to use SSL/TLS over the transport layer? Would you mind rephrasing the question please?

You can very well use a PKCS#12 store as your Transport client's keystore by setting

.put("xpack.security.http.ssl.keystore.path", "path/to/your/PKCS12")
.put("xpack.security.http.ssl.keystore.password", xxx)
15

Now I'm confuse, shouldn't I be using xpack.security.transport.ssl instead of xpack.security.http.ssl for the TransportClient ?

16

As far as I know, the Elasticsearch-Hadoop plugin uses the HTTP interface and not the transport client.

17

Apologies ,that was a copy paste error. It's the transport settings you need to set. I'll update the original post

18

In fact, I tried with :

.put("xpack.security.transport.ssl.keystore.path", "the/path") 
.put("xpack.security.transport.ssl.keystore.password", "password")

but it doesn't work either, I got the same error as above :thinking:

I'm pretty sure I missed something but I don't know what ...

19

Please share the whole example code that you are using so that we can see all the settings. Also, what is in the keystore that you are using as a value for xpack.security.transport.ssl.keystore.path ? Does it contain a key and certificate entry? How did you create it ?

Can you share the output of

openssl pkcs12 -info -in /path/to/your.p12 -noout
21

Here is my code :

public org.elasticsearch.client.Client create() throws Exception {
			org.elasticsearch.common.settings.Settings settings = org.elasticsearch.common.settings.Settings.builder()
                    .put("cluster.name", "elastic_cluster6")
					.put("xpack.security.transport.ssl.enabled", "true") 
					.put("xpack.security.transport.ssl.truststore.path", "/home/keystore/es6.p12")
					.put("xpack.security.transport.ssl.truststore.password", xxx)
					.put("xpack.ssl.verification_mode", "certificate")
					.build();


			org.elasticsearch.client.transport.TransportClient transportClient = new org.elasticsearch.xpack.client.PreBuiltXPackTransportClient(settings);


			transportClient.addTransportAddress(new org.elasticsearch.common.transport.TransportAddress(xxx));


			return transportClient;
		}

For the x-pack configuration I got :

xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.type: PKCS12
xpack.security.transport.ssl.keystore.path: certs/elastic-certificates.p12
xpack.security.transport.ssl.truststore.type: PKCS12
xpack.security.transport.ssl.truststore.path: certs/elastic-certificates.p12

xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.keystore.path: certs/elastic-certificates.p12
xpack.security.http.ssl.truststore.path: certs/elastic-certificates.p12

xpack.security.audit.enabled: true
xpack.security.authc.anonymous.roles: superuser

And here is the content of the elastic-certificates.p12 :

MAC Iteration 100000
MAC verified OK
PKCS7 Data
Shrouded Keybag: pbeWithSHA1And3-KeyTripleDES-CBC, Iteration 50000
PKCS7 Encrypted data: pbeWithSHA1And40BitRC2-CBC, Iteration 50000
Certificate bag
Certificate bag