Problem Statement - Unable to transfer filebeat logs to logstash when SSL is enabled. Logstash keeps on complaining about incorrect OpenSSL version number
PS - Filebeat works fine when SSL is disabled & able to transfer documents to elastic via logstash
Options tried so far -
- set ssl_verify_mode => "peer" on logstash input beats
- version check for both filebeat and logstash(7.5.2)
- Beats plugin check 6.0.5
- Beats plugin update. This crashes logstash when updated to 6.1.0. Its currently tracked under a separate topic
- Connect directly using openssl from filebeat to logstash. Its works fine and uses cipher TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 for SSL connection
- Turned off SSL certificate verification(for self-signed certs)
I would appreciate if someone can really help me figure out what im missing to resolve this issue.
Cert Type - Self-Signed
Filebeat Version - 7.5.2
Filebeat Error -
output.go:92: ERR Failed to publish events: client is not connected
output.go:92: ERR Failed to publish events: client is not connected
async.go:235: ERR Failed to publish events caused by: client is not connected
Filebeat Config
filebeat.config.modules.path: ${path.config}/modules.d/*.yml
output.logstash:
hosts: ["logstash.example.com:50499"]
ssl.enabled: true
ssl.certificate_authorities: ["/etc/pki/tls/certs/logstash-ca.crt"]
ssl.certificate: "/etc/pki/tls/certs/logstash.pem"
ssl.key: "/etc/pki/tls/certs/logstash.pkcs8.key"
Filebeat Test
logstash: logstash.example.com:50499...
connection...
parse host... OK
dns lookup... OK
addresses: 10.XX.XX.XX
dial up... OK
TLS...
security: server's certificate chain verification is enabled
handshake... OK
TLS version: TLSv1.2
dial up... OK
talk to server... OK
Logstash Version - 7.5.2
Logstash Error -
io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: error:100000f7:SSL routines:OPENSSL_internal:WRONG_VERSION_NUMBER
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:463) ~[netty-all-4.1.44.Final.jar:4.1.44.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:271) ~[netty-all-4.1.44.Final.jar:4.1.44.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377) ~[netty-all-4.1.44.Final.jar:4.1.44.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) ~[netty-all-4.1.44.Final.jar:4.1.44.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355) ~[netty-all-4.1.44.Final.jar:4.1.44.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[netty-all-4.1.44.Final.jar:4.1.44.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377) ~[netty-all-4.1.44.Final.jar:4.1.44.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) ~[netty-all-4.1.44.Final.jar:4.1.44.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[netty-all-4.1.44.Final.jar:4.1.44.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[netty-all-4.1.44.Final.jar:4.1.44.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[netty-all-4.1.44.Final.jar:4.1.44.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650) ~[netty-all-4.1.44.Final.jar:4.1.44.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576) ~[netty-all-4.1.44.Final.jar:4.1.44.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[netty-all-4.1.44.Final.jar:4.1.44.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [netty-all-4.1.44.Final.jar:4.1.44.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-all-4.1.44.Final.jar:4.1.44.Final]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-all-4.1.44.Final.jar:4.1.44.Final]
at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: javax.net.ssl.SSLHandshakeException: error:100000f7:SSL routines:OPENSSL_internal:WRONG_VERSION_NUMBER
at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.sslReadErrorResult(ReferenceCountedOpenSslEngine.java:1260) ~[netty-all-4.1.44.Final.jar:4.1.44.Final]
at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1221) ~[netty-all-4.1.44.Final.jar:4.1.44.Final]
at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1292) ~[netty-all-4.1.44.Final.jar:4.1.44.Final]
at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1335) ~[netty-all-4.1.44.Final.jar:4.1.44.Final]
at io.netty.handler.ssl.SslHandler$SslEngineType$1.unwrap(SslHandler.java:205) ~[netty-all-4.1.44.Final.jar:4.1.44.Final]
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1324) ~[netty-all-4.1.44.Final.jar:4.1.44.Final]
at io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1231) ~[netty-all-4.1.44.Final.jar:4.1.44.Final]
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1268) ~[netty-all-4.1.44.Final.jar:4.1.44.Final]
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:493) ~[netty-all-4.1.44.Final.jar:4.1.44.Final]
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:432) ~[netty-all-4.1.44.Final.jar:4.1.44.Final]
Logstash Beats Plugin Version - 6.0.5
Logstash input Config -
input {
beats {
port => 50499
ssl => true
ssl_certificate_authorities => ["/etc/pki/tls/certs/logstash-ca.crt"]
ssl_certificate => '/etc/pki/tls/certs/logstash.pem'
ssl_key => '/etc/pki/tls/certs/logstash.pkcs8.key'
ssl_verify_mode => "peer"
}
}