QUESTION
I have installed Shield in my Elasticsearch cluster, and configured both Transport and HTTP protocol to use SSL.
Everything appears to be working ok, but I am getting these exceptions in the elasticsearch log:
[2015-01-21 10:13:10,239][WARN ][shield.transport.netty ] [Node123] exception caught on transport layer [[id: 0x7fb623b9, /192.168.0.23:61692 => /192.168.0.12:9300]], closing connection
org.elasticsearch.common.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 455300000061000000000000000000000f42a312636c75737465722f6e6f6465732f696e666f010a0100000d417574686f72697a6174696f6e001a4261736963205a47467a614470696157646d6247467763773d3d01065f6c6f63616c00010101010101010101
at org.elasticsearch.common.netty.handler.ssl.SslHandler.decode(SslHandler.java:923)
at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:425)
at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
at org.elasticsearch.common.netty.handler.ipfilter.IpFilteringHandlerImpl.handleUpstream(IpFilteringHandlerImpl.java:154)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:268)
at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:255)
at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
[2015-01-21 10:13:10,240][ERROR][shield.transport.netty ] [Node123] SSL / TLS handshake failed, closing channel: null
What is happening?
ANSWER
When SSL is enabled for HTTP or Transport protocols (or both) and a clear text HTTP request will hit the node, the above stacktrace will be printed.
This specific stacktrace is related to a clear text HTTP request hitting the transport port (see port target port of request is 9300), most likely in this scenario one of the nodes is still running with configuration using clear text rather than SSL.
Same thing would happen should a clear text request hit a node HTTP port, e.g. if you were to open your browser and use http://your_node:9200 instead of https://your_node:9200.
To fix this, make sure all the nodes, clients and browsers interacting with your Secure cluster are issuing HTTP SSL requests.