NO_SHARED_CIPHER error when beats is behind AWS load balancer

smilin_stan · August 16, 2018, 3:56pm

I am running logstash 6.3.2 on Amazon Linux 2. My beats input is as follows:

input {
  beats {
	port => 5044
	ssl => true
	ssl_certificate => "/etc/ssl/certs/logstash.crt"
	ssl_key => "/etc/ssl/certs/logstash.pem"
  }
}

In front of the EC2 instance is an elastic load balancer which has an SSL listener on port 5044. The listener is terminated on the EC2 instance on port 5044 using SSL.

The certificate on the load balancer is provided by Amazon. The certificate being used by logstash is self-signed.

I have a separate server on which I am running metricbeat. My metricbeat.yml is as follows:

metricbeat.config.modules:
  path: ${path.config}/modules.d/*.yml

output.logstash:
  hosts: [ "my.load.balancer:5044" ]
  ssl.enabled: true

When I run metricbeat -c metricbeat.yml -e -v I see the following errors:

2018-08-16T16:35:59.480+0100	ERROR	logstash/async.go:235	Failed to publish events caused by: read tcp 10.5.28.114:54690->10.5.250.141:5044: i/o timeout
2018-08-16T16:35:59.480+0100	ERROR	logstash/async.go:235	Failed to publish events caused by: read tcp 10.5.28.114:54690->10.5.250.141:5044: i/o timeout
2018-08-16T16:35:59.481+0100	ERROR	logstash/async.go:235	Failed to publish events caused by: read tcp 10.5.28.114:54690->10.5.250.141:5044: i/o timeout
2018-08-16T16:35:59.481+0100	INFO	[publish]	pipeline/retry.go:149	retryer: send wait signal to consumer
2018-08-16T16:35:59.481+0100	INFO	[publish]	pipeline/retry.go:151	  done
2018-08-16T16:35:59.483+0100	ERROR	logstash/async.go:235	Failed to publish events caused by: client is not connected
2018-08-16T16:36:00.483+0100	ERROR	pipeline/output.go:92	Failed to publish events: client is not connected
2018-08-16T16:36:00.510+0100	INFO	[publish]	pipeline/retry.go:172	retryer: send unwait-signal to consumer
2018-08-16T16:36:00.510+0100	INFO	[publish]	pipeline/retry.go:174	  done

In the logstash logs there is the following error:

[2018-08-16T16:37:02,576][INFO ][org.logstash.beats.BeatsHandler] [local: 0.0.0.0:5044, remote: 10.5.250.141:47332] Handling exception: javax.net.ssl.SSLHandshakeException: error:100000b8:SSL routines:OPENSSL_internal:NO_SHARED_CIPHER
[2018-08-16T16:37:02,577][WARN ][io.netty.channel.DefaultChannelPipeline] An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.
io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: error:100000b8:SSL routines:OPENSSL_internal:NO_SHARED_CIPHER
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:459) ~[logstash-input-tcp-5.0.9.jar:?]
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265) ~[logstash-input-tcp-5.0.9.jar:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) ~[logstash-input-tcp-5.0.9.jar:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) ~[logstash-input-tcp-5.0.9.jar:?]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) ~[logstash-input-tcp-5.0.9.jar:?]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359) ~[logstash-input-tcp-5.0.9.jar:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) ~[logstash-input-tcp-5.0.9.jar:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) ~[logstash-input-tcp-5.0.9.jar:?]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935) ~[logstash-input-tcp-5.0.9.jar:?]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:141) ~[logstash-input-tcp-5.0.9.jar:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645) ~[logstash-input-tcp-5.0.9.jar:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580) ~[logstash-input-tcp-5.0.9.jar:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497) ~[logstash-input-tcp-5.0.9.jar:?]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) [logstash-input-tcp-5.0.9.jar:?]
	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [logstash-input-tcp-5.0.9.jar:?]
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [logstash-input-tcp-5.0.9.jar:?]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]

If I remove the SSL config from the beats input:

input {
  beats {
	port => 5044
  }
}

And change the termination of the ELB listener from SSL to TCP, everything works fine. This implies the issue lies with the SSL configuration between the load balancer and logstash.

If I re-add the SSL config to the beats input and change the metricbeat.yml to connect to the logstash EC2 directly like so:

metricbeat.config.modules:
  path: ${path.config}/modules.d/*.yml

output.logstash:
  hosts: [ "my.ec2.server:5044" ]
  ssl.enabled: true
  ssl.verification_mode: none    (to get round the self-signed certificate)

This config also works fine, which indicates that the logstash config is fine and it must be something to do with the SSL between ELB and logstash. To summarise:

					SSL                   SSL
Metricbeat Server ------> Load Balancer ------> Logstash (Beats SSL)		DOES NOT WORK


					SSL                   TCP
Metricbeat Server ------> Load Balancer ------> Logstash (Beats Non SSL)	WORKS


					SSL
Metricbeat Server ------> Logstash (Beats SSL)								WORKS

Having spent two days fighting this I'm out of ideas, does anyone have any pointers?

kvch · August 17, 2018, 9:37am

Is the certificate of the ELB instance is trusted by Logstash?

smilin_stan · August 17, 2018, 3:36pm

The beats input plugin has a setting called ssl_verify_mode which defaults to none. I tried setting it manually to none but the problem persists.

The certificate on the ELB is provided by Amazon and is trusted by the JVM trust store. The error mentions ciphers too, whereas a certificate trust failure should result in a different error message I think?

smilin_stan · August 23, 2018, 5:30pm

So for anyone else encountering the same problem, I had to manually set the ciphers on the beats plugin as the default ciphers were not supported by AWS ELB. My input now looks like:

input {
  beats {
	port => 5044
	ssl => true
	ssl_certificate => "/etc/ssl/certs/logstash.crt"
	ssl_key => "/etc/ssl/certs/logstash.pem"
	cipher_suites => [ "TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384","TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA","TLS_RSA_WITH_AES_256_CBC_SHA","TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256","TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA","TLS_RSA_WITH_AES_128_CBC_SHA" ]
  }
}

This is working fine.

system · September 20, 2018, 7:30pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
A big trouble about from filebeat to logstash use SSL to transition Beats filebeat	6	1348	May 18, 2017
Filebeat 6.1.1 not connecting to logstash over ssl Beats filebeat	10	5369	February 20, 2018
Logs are not being pushed from beats to logstash Beats	6	2687	August 26, 2016
Cannot setup SSL for Filebeat to Logstash Beats	9	2295	February 21, 2018
Debug Errors while setting up SSL communication between Filebeat and Logstash Beats filebeat	7	7355	December 15, 2017

NO_SHARED_CIPHER error when beats is behind AWS load balancer

Related topics