NO_SHARED_CIPHER error when beats is behind AWS load balancer

Elastic Stack Beats
1

I am running logstash 6.3.2 on Amazon Linux 2. My beats input is as follows:

input {
  beats {
	port => 5044
	ssl => true
	ssl_certificate => "/etc/ssl/certs/logstash.crt"
	ssl_key => "/etc/ssl/certs/logstash.pem"
  }
}

In front of the EC2 instance is an elastic load balancer which has an SSL listener on port 5044. The listener is terminated on the EC2 instance on port 5044 using SSL.

The certificate on the load balancer is provided by Amazon. The certificate being used by logstash is self-signed.

I have a separate server on which I am running metricbeat. My metricbeat.yml is as follows:

metricbeat.config.modules:
  path: ${path.config}/modules.d/*.yml

output.logstash:
  hosts: [ "my.load.balancer:5044" ]
  ssl.enabled: true

When I run metricbeat -c metricbeat.yml -e -v I see the following errors:

2018-08-16T16:35:59.480+0100	ERROR	logstash/async.go:235	Failed to publish events caused by: read tcp 10.5.28.114:54690->10.5.250.141:5044: i/o timeout
2018-08-16T16:35:59.480+0100	ERROR	logstash/async.go:235	Failed to publish events caused by: read tcp 10.5.28.114:54690->10.5.250.141:5044: i/o timeout
2018-08-16T16:35:59.481+0100	ERROR	logstash/async.go:235	Failed to publish events caused by: read tcp 10.5.28.114:54690->10.5.250.141:5044: i/o timeout
2018-08-16T16:35:59.481+0100	INFO	[publish]	pipeline/retry.go:149	retryer: send wait signal to consumer
2018-08-16T16:35:59.481+0100	INFO	[publish]	pipeline/retry.go:151	  done
2018-08-16T16:35:59.483+0100	ERROR	logstash/async.go:235	Failed to publish events caused by: client is not connected
2018-08-16T16:36:00.483+0100	ERROR	pipeline/output.go:92	Failed to publish events: client is not connected
2018-08-16T16:36:00.510+0100	INFO	[publish]	pipeline/retry.go:172	retryer: send unwait-signal to consumer
2018-08-16T16:36:00.510+0100	INFO	[publish]	pipeline/retry.go:174	  done

In the logstash logs there is the following error:

[2018-08-16T16:37:02,576][INFO ][org.logstash.beats.BeatsHandler] [local: 0.0.0.0:5044, remote: 10.5.250.141:47332] Handling exception: javax.net.ssl.SSLHandshakeException: error:100000b8:SSL routines:OPENSSL_internal:NO_SHARED_CIPHER
[2018-08-16T16:37:02,577][WARN ][io.netty.channel.DefaultChannelPipeline] An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.
io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: error:100000b8:SSL routines:OPENSSL_internal:NO_SHARED_CIPHER
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:459) ~[logstash-input-tcp-5.0.9.jar:?]
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265) ~[logstash-input-tcp-5.0.9.jar:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) ~[logstash-input-tcp-5.0.9.jar:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) ~[logstash-input-tcp-5.0.9.jar:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) ~[logstash-input-tcp-5.0.9.jar:?]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359) ~[logstash-input-tcp-5.0.9.jar:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) ~[logstash-input-tcp-5.0.9.jar:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) ~[logstash-input-tcp-5.0.9.jar:?]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935) ~[logstash-input-tcp-5.0.9.jar:?]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:141) ~[logstash-input-tcp-5.0.9.jar:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645) ~[logstash-input-tcp-5.0.9.jar:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580) ~[logstash-input-tcp-5.0.9.jar:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497) ~[logstash-input-tcp-5.0.9.jar:?]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) [logstash-input-tcp-5.0.9.jar:?]
	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [logstash-input-tcp-5.0.9.jar:?]
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [logstash-input-tcp-5.0.9.jar:?]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]

If I remove the SSL config from the beats input:

input {
  beats {
	port => 5044
  }
}

And change the termination of the ELB listener from SSL to TCP, everything works fine. This implies the issue lies with the SSL configuration between the load balancer and logstash.

If I re-add the SSL config to the beats input and change the metricbeat.yml to connect to the logstash EC2 directly like so:

metricbeat.config.modules:
  path: ${path.config}/modules.d/*.yml

output.logstash:
  hosts: [ "my.ec2.server:5044" ]
  ssl.enabled: true
  ssl.verification_mode: none    (to get round the self-signed certificate)

This config also works fine, which indicates that the logstash config is fine and it must be something to do with the SSL between ELB and logstash. To summarise:

					SSL                   SSL
Metricbeat Server ------> Load Balancer ------> Logstash (Beats SSL)		DOES NOT WORK


					SSL                   TCP
Metricbeat Server ------> Load Balancer ------> Logstash (Beats Non SSL)	WORKS


					SSL
Metricbeat Server ------> Logstash (Beats SSL)								WORKS

Having spent two days fighting this I'm out of ideas, does anyone have any pointers?

2

Is the certificate of the ELB instance is trusted by Logstash?

3

The beats input plugin has a setting called ssl_verify_mode which defaults to none. I tried setting it manually to none but the problem persists.

The certificate on the ELB is provided by Amazon and is trusted by the JVM trust store. The error mentions ciphers too, whereas a certificate trust failure should result in a different error message I think?

4

So for anyone else encountering the same problem, I had to manually set the ciphers on the beats plugin as the default ciphers were not supported by AWS ELB. My input now looks like:

input {
  beats {
	port => 5044
	ssl => true
	ssl_certificate => "/etc/ssl/certs/logstash.crt"
	ssl_key => "/etc/ssl/certs/logstash.pem"
	cipher_suites => [ "TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384","TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA","TLS_RSA_WITH_AES_256_CBC_SHA","TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256","TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA","TLS_RSA_WITH_AES_128_CBC_SHA" ]
  }
}

This is working fine.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.