Cannot retrieve remote IP address

jordyd · April 16, 2018, 3:00pm

I created certificates using the x-pack certutil, but Logstash is giving me this error. I created a certificate with an IP address for Filebeat and Logstash because I want Filebeat to send data to Logstash which is installed on a different server.

[2018-04-12T12:51:21,323][INFO ][org.logstash.beats.BeatsHandler] [local: 0.0.0.0:5044, remote: 10.203.207.127:50626] Handling exception: javax.net.ssl.SSLHandshakeException: error:100000f7:SSL routines:OPENSSL_internal:WRONG_VERSION_NUMBER
[2018-04-12T12:51:22,322][DEBUG][org.logstash.netty.SslSimpleBuilder] Available ciphers:[ECDHE-ECDSA-AES128-GCM-SHA256, ECDHE-RSA-AES128-GCM-SHA256, ECDHE-ECDSA-AES256-GCM-SHA384, ECDHE-RSA-AES256-GCM-SHA384, ECDHE-ECDSA-CHACHA20-POLY1305, ECDHE-RSA-CHACHA20-POLY1305, ECDHE-PSK-CHACHA20-POLY1305, ECDHE-ECDSA-AES128-SHA, ECDHE-ECDSA-AES128-SHA256, ECDHE-RSA-AES128-SHA, ECDHE-RSA-AES128-SHA256, ECDHE-PSK-AES128-CBC-SHA, ECDHE-ECDSA-AES256-SHA, ECDHE-ECDSA-AES256-SHA384, ECDHE-RSA-AES256-SHA, ECDHE-RSA-AES256-SHA384, ECDHE-PSK-AES256-CBC-SHA, AES128-GCM-SHA256, AES256-GCM-SHA384, AES128-SHA, AES128-SHA256, PSK-AES128-CBC-SHA, AES256-SHA, AES256-SHA256, PSK-AES256-CBC-SHA, DES-CBC3-SHA]
[2018-04-12T12:51:22,322][DEBUG][org.logstash.netty.SslSimpleBuilder] Ciphers:  [TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256]
[2018-04-12T12:51:22,322][DEBUG][org.logstash.netty.SslSimpleBuilder] Certificate Authorities: [/etc/logstash/cert/ca.crt]
[2018-04-12T12:51:22,322][DEBUG][org.logstash.netty.SslSimpleBuilder] Load certificates collection
[2018-04-12T12:51:22,327][DEBUG][org.logstash.netty.SslSimpleBuilder] Loading certificates from file /etc/logstash/cert/ca.crt
[2018-04-12T12:51:22,341][DEBUG][org.logstash.netty.SslSimpleBuilder] TLS: [TLSv1, TLSv1.1, TLSv1.2]
[2018-04-12T12:51:22,357][DEBUG][logstash.inputs.beats    ] Cannot retrieve remote IP address for beats input - remoteAddress is nil
[2018-04-12T12:51:22,359][DEBUG][org.logstash.beats.BeatsHandler] [local: 0.0.0.0:5044, remote: undefined] Handling exception: javax.net.ssl.SSLHandshakeException: error:100000f7:SSL routines:OPENSSL_internal:WRONG_VERSION_NUMBER
io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: error:100000f7:SSL routines:OPENSSL_internal:WRONG_VERSION_NUMBER
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:459) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:141) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-all-4.1.18.Final.jar:4.1.18.Final]
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-all-4.1.18.Final.jar:4.1.18.Final]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_151]
Caused by: javax.net.ssl.SSLHandshakeException: error:100000f7:SSL routines:OPENSSL_internal:WRONG_VERSION_NUMBER
	at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.sslReadErrorResult(ReferenceCountedOpenSslEngine.java:1120) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
	at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1080) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
	at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1146) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
	at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1189) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
	at io.netty.handler.ssl.SslHandler$SslEngineType$1.unwrap(SslHandler.java:216) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
	at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1248) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
	at io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1171) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
	at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1196) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
	... 16 more
[2018-04-12T12:51:22,360][INFO ][org.logstash.beats.BeatsHandler] [local: 0.0.0.0:5044, remote: undefined] Handling exception: javax.net.ssl.SSLHandshakeException: error:100000f7:SSL routines:OPENSSL_internal:WRONG_VERSION_NUMBER

Yogesh_Gaikwad · April 17, 2018, 1:10am

Looks like TLS version support mismatch? Can you please check what protocol versions have been enabled on logstash and beats side?

jordyd · April 17, 2018, 7:13am

Where can I find the protocol versions?

Yogesh_Gaikwad · April 23, 2018, 1:03am

Hi @jordyd,

Sorry, I missed this question from you.
Please find following links for TLS configuration relating to version and cipher suite configs on beats input plugin:

Also look through this issue on logstash input beats plugin and see if there is any similarity with your setup

github.com/logstash-plugins/logstash-input-beats

Unable to get ssl working between Filebeat and Logstash

opened 11:00AM - 01 Feb 18 UTC

closed 09:15PM - 23 Feb 18 UTC

lukos

History [here](https://discuss.elastic.co/t/cannot-setup-ssl-for-filebeat-to-log…stash/117785) including some of the debug output and the original full error. I have a working filebeat to logstash pipeline with NO ssl. I wanted to enable SSL but I cannot get it to work and it is not completely clear why. Filebeat version 6.1 for Windows 10 Logstash version 6.1 on Ubuntu 16:04 Curl running from Ubuntu 16:04 ``` input { beats { port => "5044" ssl => true ssl_certificate => "/etc/ssl/certs/wildcard4-combined.cer" ssl_key => "/etc/ssl/private/wildcard4.pem" } } ``` The cert is a commercial cert chained to a well-known root (curl is happy with it!). No changes are made in filebeat.yml from the non-ssl since I am not using client auth and have used the same port number. Filebeat shows the following error when I try and run it over ssl: "async.go:235: ERR Failed to publish events caused by: lumberjack protocol error" If I run curl -v from another machine as suggested in setting up SSL, I get the following error: "ALPN, server did not agree to a protocol" even though the logstash debug doesn't log any error other than invalid frame (which I would expect the curl test to cause?). This is where it gets confusing! Note all the spelling differences, some expected, some not. Curl reports on the console that it is using ECDHE_RSA_AES_256_GCM_SHA384 however, this is not listed in nss.c, which instead says it supports TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (I thought that curl used nss but that might not be true). Gnu DOES support the named cipher but it has it prefixed with TLS_ according to [here](https://wiki.mozilla.org/Security/Server_Side_TLS). * Logstash reports that it has a mapping for TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 => ECDHE-RSA-AES256-GCM-SHA384 * TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 is listed as "Cipher is supported" in LS * ECDHE-RSA-AES256-GCM-SHA384 is listed in LS "Available ciphers" when a connection is made. * TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 is listed next to "Ciphers" in LS when the connection is made, * Logstash reports HANDSHAKEN: TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, which sounds like it should have worked. The only thing I can possibly think right now is that in this plethora of slightly different named ciphers, there is a mapping missing or a typo? It is also possible that one of the applications in this set has been upgraded to only support some newer protocols even though the others have not?

Hope this helps !

Regards,
Yogesh

system · May 21, 2018, 1:03am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.