Filebeat - Logstash SSL not working

Problem Statement - Unable to transfer filebeat logs to logstash when SSL is enabled. Logstash keeps on complaining about incorrect OpenSSL version number

PS - Filebeat works fine when SSL is disabled & able to transfer documents to elastic via logstash

Options tried so far -

  1. set ssl_verify_mode => "peer" on logstash input beats
  2. version check for both filebeat and logstash(7.5.2)
  3. Beats plugin check 6.0.5
  4. Beats plugin update. This crashes logstash when updated to 6.1.0. Its currently tracked under a separate topic
  5. Connect directly using openssl from filebeat to logstash. Its works fine and uses cipher TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 for SSL connection
  6. Turned off SSL certificate verification(for self-signed certs)

I would appreciate if someone can really help me figure out what im missing to resolve this issue.

Cert Type - Self-Signed
Filebeat Version - 7.5.2
Filebeat Error -

output.go:92: ERR Failed to publish events: client is not connected
output.go:92: ERR Failed to publish events: client is not connected
async.go:235: ERR Failed to publish events caused by: client is not connected

Filebeat Config

filebeat.config.modules.path: ${path.config}/modules.d/*.yml
output.logstash:
  hosts: ["logstash.example.com:50499"]
  ssl.enabled: true
  ssl.certificate_authorities: ["/etc/pki/tls/certs/logstash-ca.crt"]
  ssl.certificate: "/etc/pki/tls/certs/logstash.pem"
  ssl.key: "/etc/pki/tls/certs/logstash.pkcs8.key"

Filebeat Test

        logstash: logstash.example.com:50499...
        connection...
        parse host... OK
        dns lookup... OK
        addresses: 10.XX.XX.XX
       dial up... OK
       TLS...
       security: server's certificate chain verification is enabled
       handshake... OK
       TLS version: TLSv1.2
      dial up... OK
      talk to server... OK

Logstash Version - 7.5.2
Logstash Error -

io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: error:100000f7:SSL routines:OPENSSL_internal:WRONG_VERSION_NUMBER
     at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:463) ~[netty-all-4.1.44.Final.jar:4.1.44.Final]
     at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:271) ~[netty-all-4.1.44.Final.jar:4.1.44.Final]
     at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377) ~[netty-all-4.1.44.Final.jar:4.1.44.Final]
     at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) ~[netty-all-4.1.44.Final.jar:4.1.44.Final]
     at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355) ~[netty-all-4.1.44.Final.jar:4.1.44.Final]
     at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[netty-all-4.1.44.Final.jar:4.1.44.Final]
     at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377) ~[netty-all-4.1.44.Final.jar:4.1.44.Final]
     at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) ~[netty-all-4.1.44.Final.jar:4.1.44.Final]
     at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[netty-all-4.1.44.Final.jar:4.1.44.Final]
     at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[netty-all-4.1.44.Final.jar:4.1.44.Final]
     at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[netty-all-4.1.44.Final.jar:4.1.44.Final]
     at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650) ~[netty-all-4.1.44.Final.jar:4.1.44.Final]
     at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576) ~[netty-all-4.1.44.Final.jar:4.1.44.Final]
     at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[netty-all-4.1.44.Final.jar:4.1.44.Final]
     at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [netty-all-4.1.44.Final.jar:4.1.44.Final]
     at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-all-4.1.44.Final.jar:4.1.44.Final]
     at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-all-4.1.44.Final.jar:4.1.44.Final]
     at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: javax.net.ssl.SSLHandshakeException: error:100000f7:SSL routines:OPENSSL_internal:WRONG_VERSION_NUMBER
     at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.sslReadErrorResult(ReferenceCountedOpenSslEngine.java:1260) ~[netty-all-4.1.44.Final.jar:4.1.44.Final]
     at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1221) ~[netty-all-4.1.44.Final.jar:4.1.44.Final]
     at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1292) ~[netty-all-4.1.44.Final.jar:4.1.44.Final]
     at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1335) ~[netty-all-4.1.44.Final.jar:4.1.44.Final]
     at io.netty.handler.ssl.SslHandler$SslEngineType$1.unwrap(SslHandler.java:205) ~[netty-all-4.1.44.Final.jar:4.1.44.Final]
     at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1324) ~[netty-all-4.1.44.Final.jar:4.1.44.Final]
     at io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1231) ~[netty-all-4.1.44.Final.jar:4.1.44.Final]
     at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1268) ~[netty-all-4.1.44.Final.jar:4.1.44.Final]
     at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:493) ~[netty-all-4.1.44.Final.jar:4.1.44.Final]
     at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:432) ~[netty-all-4.1.44.Final.jar:4.1.44.Final]

Logstash Beats Plugin Version - 6.0.5
Logstash input Config -

      input {
      beats {
        port => 50499
        ssl => true
        ssl_certificate_authorities => ["/etc/pki/tls/certs/logstash-ca.crt"]
        ssl_certificate => '/etc/pki/tls/certs/logstash.pem'
        ssl_key => '/etc/pki/tls/certs/logstash.pkcs8.key'
        ssl_verify_mode => "peer"
      }
        }

This issue stands resolved. The problem was with our edge load balancer which was adding an additional layer of SSL encryption which was causing the OpenSSL wrong version error.

Once additional SSL was disabled, filebeat was able to transfer logs to elasticsearch via logstash successfully.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.