ElasticSearch 6.5.1 TLS setting

Elastic Stack Elasticsearch
1

Hi ,

i am struggling for long time to establish TLS communication between 2 nodes.
I followed : https://www.elastic.co/guide/en/elasticsearch/reference/6.5/configuring-tls.html#node-certificates
Also check several links , nothing is working in my case .

Done:

  1. bin/elasticsearch-certutil ca
  2. bin/elasticsearch-certutil cert --ca elastic_cert_v1.p12

i've placed the created "elastic_cert_v1.p12" to each node under config/certs
Restarted both ES nodes.

Cluster is working fine without TSL.

[**Getting error:**
[WARN ][o.e.x.s.t.n.SecurityNetty4ServerTransport] [hostname] exception caught on transport layer [NettyTcpChannel{localAddress=0.0.0.0/0.0.0.0:9300, remoteAddress=/127.0.0.1:35222}], closing connection
io.netty.handler.codec.DecoderException: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 455300000027000000000000a32208004c4ba3000016696e7465726e616c3a7463702f68616e647368616b6500
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:472) ~[netty-codec-4.1.30.Final.jar:4.1.30.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278) ~[netty-codec-4.1.30.Final.jar:4.1.30.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:544) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458) [netty-transport-4.1.30.Final.jar:4.1.30.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:897) [netty-common-4.1.30.Final.jar:4.1.30.Final]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_251]
Caused by: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 455300000027000000000000a32208004c4ba3000016696e7465726e616c3a7463702f68616e647368616b6500
        at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1178) ~[netty-handler-4.1.30.Final.jar:4.1.30.Final]
        at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1243) ~[netty-handler-4.1.30.Final.jar:4.1.30.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502) ~[netty-codec-4.1.30.Final.jar:4.1.30.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441) ~[netty-codec-4.1.30.Final.jar:4.1.30.Final]
*        ... 15 more`

**elasticsearch.yml **
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

node.name: hostname
path.data: /var/apps/elasticsearch/6.5.1/data
cluster.name: elasticsearch-app_prod
network.host: 0.0.0.0

script.painless.regex.enabled: true
path.logs: /var/log/elasticsearch
http.port: 9200
path.repo: ["/var/apps/elasticsearch/6.5.1/snapshots"]

node.master: true
node.data: true
discovery.zen.ping.unicast.hosts: ["ip_node1",  "ip_node2" ]
discovery.zen.minimum_master_nodes: 1

xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path:    certs/elastic_cert_v1.p12
xpack.security.transport.ssl.truststore.path:  certs/elastic_cert_v1.p12

On second node in log there is not above error .Instead the last line is:
Cluster health status changed from [YELLOW] to [GREEN]

Any help would be greatly appreciated !
`

2

One of your nodes isn't configured to use SSL.
Did you set

xpack.security.transport.ssl.enabled: true

on every node and then restart them all?

3

yes, i have same xpack lines on both ES.yml.

4

Could you share your full elasticsearch.yml and format your codes and errors under preformatted text </> or backticks (```) as it is really hard to read otherwise.

5

looks much readable now :slight_smile: . Config yml content is same on second node.

6

As Tim mentioned, there is an ES node in the same host (remoteAddress=/127.0.0.1 ) as the one where you get the error that is not configured to use TLS for the transport layer and is attempting to connect to your node.

455300000027000000000000a32208004c4ba3000016696e7465726e616c3a7463702f68616e647368616b6500

is an attempt by that node to start a TCP handshake with your node.

7

I have only one ES running according : ps aux | grep "elastic"* (same outcome on both nodes)

Questions:

  1. How to verify TSL is properly running on each node ?

  2. Can the problem with TSL, not to be configured on node1 (where error is recorded), be with "p12" file ?
    The "elastic_cert_v1.p12" was created on one node and then copied to the second one. Could this be an issue?

  3. On second node , the log says : cluster is Green ...
    Does it mean that cluster is working with TSL ?

I'm a newbie in ES as such ..could figure out the cluster set up ,but TSL is over me at the moment.