Logstash to Logstash with Lumbejack to Beats and SSL issue

Hi,

I'm trying to connect two Logsatsh with Lumberjack output plugin and Beats input plugin But I'm getting an error on ssl verify... Let's give more context !

First I'm using Logstash 8.5 on both side (I also tried with 8.17).

To begin I have followed this documentation: Logstash-to-Logstash: Lumberjack output to Beats input | Logstash Reference [8.17] | Elastic It worked pretty well, but in production environments I would like to add certificate from my infrastructure CA. So I generated a certificate, like as in the doc but signed by my CA. Then I was ready to start the Logstash "server" (the one with beats input) with the following command:

logstash -e 'input { beats { port => 5044 ssl_certificate => "/usr/share/lumberjack/config/certificates/server.cert" ssl => "true" ssl_key => "/usr/share/lumberjack/config/certificates/server.key" codec => "json" ssl_certificate_authorities => ["/usr/share/lumberjack/config/certificates/ca.cert"] ssl_verify_mode => "force_peer" } }'

Then I try to send event with the other Logstash:

logstash -e 'input { generator { count => 5 } } output { lumberjack { codec => "json" hosts => "ls-server.elastic-system.svc.cluster.local" ssl_certificate => "/usr/share/lumberjack/config/certificates/server.cert" port => 5044 } }'

And here is the ssl error that I have (client side):

e=>#<OpenSSL::SSL::SSLError: certificate verify failed>

and server side:

io.netty.handler.codec.DecoderException: javax.net.ssl.SSLProtocolException: Unexpected content: 20
Caused by: javax.net.ssl.SSLProtocolException: Unexpected content: 20

I reproduce the behaviour in a lab with quick generated CA and certs, but still the same error.

It look like an ssl verify error because when I tried to connect with openssl it look like I pass a step:

openssl s_client -connect ls-server.elastic-system.svc.cluster.local:5044 -CAfile /usr/share/lumberjack/config/certificates/ca.cert --cert /usr/share/lumberjack/config/certificates/server.cert --key /usr/share/lumberjack/config/certificates/server.key

the error on server:

io.netty.handler.codec.DecoderException: org.logstash.beats.InvalidFrameProtocolException: Invalid version of beats protocol: 27
Caused by: org.logstash.beats.InvalidFrameProtocolException: Invalid version of beats protocol: 27
io.netty.handler.codec.DecoderException: org.logstash.beats.InvalidFrameProtocolException: Invalid version of beats protocol: 91
Caused by: org.logstash.beats.InvalidFrameProtocolException: Invalid version of beats protocol: 91

Someone I have an idea of what happened ? maybe I did something wrong ?