SSL configured in Beat but ES gives io.netty.handler.ssl.NotSslRecordException

Elastic Stack Beats
1

Hello,

I see every ~10 seconds the following elasticsearch error message:

[2018-01-13T22:38:41,877][WARN ][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [QNk3RNV] caught exception while handling client http traffic, closing connection [id: 0x1ae9cb20, L:0.0.0.0/0.0.0.0:9201 ! R:/178.62.7.19:49154]
io.netty.handler.codec.DecoderException: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 474554202f5f6e6f6465732f5f6c6f63616c2f737461747320485454502f312e310d0a486f73743a206c6730312e636c30302e646330312e696e662e6c756d6d696e6172792e746563683a393230310d0a557365722d4167656e743a20476f2d687474702d636c69656e742f312e310d0a4163636570742d456e636f64696e673a20677a69700d0a0d0a
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:459) ~[netty-codec-4.1.13.Final.jar:4.1.13.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265) ~[netty-codec-4.1.13.Final.jar:4.1.13.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:544) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.13.Final.jar:4.1.13.Final]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_151]
Caused by: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 474554202f5f6e6f6465732f5f6c6f63616c2f737461747320485454502f312e310d0a486f73743a206c6730312e636c30302e646330312e696e662e6c756d6d696e6172792e746563683a393230310d0a557365722d4167656e743a20476f2d687474702d636c69656e742f312e310d0a4163636570742d456e636f64696e673a20677a69700d0a0d0a
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1103) ~[?:?]
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489) ~[?:?]
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428) ~[?:?]
... 15 more

It looks like the communication is coming from a metricbeat instance from another server.

On the metricbeat, I have SSL connection enabled. Please see the metricbeat elasticsearch configuration:

#-------------------------- Elasticsearch output -------------------------------
output.elasticsearch:
enabled: true
hosts: ["server-hostname:9201"]
protocol: "https"
username: "elastic"
password: "somepassword"
index: "metricbeat-%{+yyyy.MM.dd}"
max_retries: -1
ssl.enabled: true
ssl.verification_mode: full
ssl.certificate_authorities: ["/etc/metricbeat/certificates/ca.crt"]
ssl.certificate: "/etc/metricbeat/certificates/certificate.crt"
ssl.key: "/etc/metricbeat/certificates/certificate.key"

If I stop the metricbeat process, the errors from the elasticsearch servers stops.

How can I figured out what is the problem?

Thank you,
F

2

What version of Metricbeat are you running? Could you please post the full Metricbeat config and some logs from Metricbeat.

Overall that section of the Metricbeat config appears correct. I don't commonly see the protocol option used (though it should work).

  hosts: ["server-hostname:9201"]
  protocol: "https"

Can you try this instead

  hosts: ["https://server-hostname:9201"]
1 Like
3

That's it. That solved it for me. protocol isn't supported by the elasticsearch module.

4

Hello Andrew,

I tried to reproduce it, but for some reason is not happening anymore. I don't know what is different. But it looks like it is working with the protocol: "https".

I will come back if I see the problem again.

Thanks,
F

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.