After 6.8.0 to 7.3.2 upgrade (no issues in 7.0 upgrade assistant in kibana) Elastic fails to start input.beats with the error below:
[2019-10-01T14:30:32,879][WARN ][org.logstash.beats.Server] Exception caught in channel initializer
javax.net.ssl.SSLException: failed to set certificate and key
at io.netty.handler.ssl.ReferenceCountedOpenSslServerContext.newSessionContext(ReferenceCountedOpenSslServerContext.java:140) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.handler.ssl.OpenSslServerContext.<init>(OpenSslServerContext.java:352) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.handler.ssl.OpenSslServerContext.<init>(OpenSslServerContext.java:335) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.handler.ssl.SslContext.newServerContextInternal(SslContext.java:422) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.handler.ssl.SslContextBuilder.build(SslContextBuilder.java:447) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
at org.logstash.netty.SslSimpleBuilder.build(SslSimpleBuilder.java:128) ~[logstash-input-beats-6.0.1.jar:?]
at org.logstash.beats.Server$BeatsInitializer.initChannel(Server.java:131) ~[logstash-input-beats-6.0.1.jar:?]
at org.logstash.beats.Server$BeatsInitializer.initChannel(Server.java:101) [logstash-input-beats-6.0.1.jar:?]
at io.netty.channel.ChannelInitializer.initChannel(ChannelInitializer.java:115) [netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.ChannelInitializer.handlerAdded(ChannelInitializer.java:107) [netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.DefaultChannelPipeline.callHandlerAdded0(DefaultChannelPipeline.java:637) [netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.DefaultChannelPipeline.access$000(DefaultChannelPipeline.java:46) [netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.DefaultChannelPipeline$PendingHandlerAddedTask.execute(DefaultChannelPipeline.java:1487) [netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.DefaultChannelPipeline.callHandlerAddedForAllHandlers(DefaultChannelPipeline.java:1161) [netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.DefaultChannelPipeline.invokeHandlerAddedIfNeeded(DefaultChannelPipeline.java:686) [netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.AbstractChannel$AbstractUnsafe.register0(AbstractChannel.java:510) [netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.AbstractChannel$AbstractUnsafe.access$200(AbstractChannel.java:423) [netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.AbstractChannel$AbstractUnsafe$1.run(AbstractChannel.java:482) [netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) [netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404) [netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:462) [netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:897) [netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-all-4.1.30.Final.jar:4.1.30.Final]
at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: java.security.KeyStoreException: Key protection algorithm not found: java.security.KeyStoreException: Certificate chain is not valid
at sun.security.pkcs12.PKCS12KeyStore.setKeyEntry(PKCS12KeyStore.java:704) ~[?:?]
at sun.security.pkcs12.PKCS12KeyStore.engineSetKeyEntry(PKCS12KeyStore.java:601) ~[?:?]
at sun.security.util.KeyStoreDelegator.engineSetKeyEntry(KeyStoreDelegator.java:111) ~[?:?]
at java.security.KeyStore.setKeyEntry(KeyStore.java:1174) ~[?:?]
at io.netty.handler.ssl.SslContext.buildKeyStore(SslContext.java:1004) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.handler.ssl.ReferenceCountedOpenSslServerContext.newSessionContext(ReferenceCountedOpenSslServerContext.java:126) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
... 23 more
Caused by: java.security.KeyStoreException: Certificate chain is not valid
at sun.security.pkcs12.PKCS12KeyStore.setKeyEntry(PKCS12KeyStore.java:651) ~[?:?]
at sun.security.pkcs12.PKCS12KeyStore.engineSetKeyEntry(PKCS12KeyStore.java:601) ~[?:?]
at sun.security.util.KeyStoreDelegator.engineSetKeyEntry(KeyStoreDelegator.java:111) ~[?:?]
My logstash setings for beats input are
beats {
port => 5044
ssl => true
ssl_certificate => "/logstash/logs.redacted.com.pem"
ssl_key => "/logstash/logs.redacted.com.key"
type => "beats"
}
I'm using x502 ssl certificate signed by GoDaddy
X509v3 Basic Constraints: critical
CA:FALSE
X509v3 Extended Key Usage:
TLS Web Server Authentication, TLS Web Client Authentication
The pem and key files set was working with previous versions incl. 6.8.0 and beats endpoint was receiving data. Elk is running under docker swarm, with configuration and cert files mounted using bind.
I wasn't able to find relevant information on this issue so tried several solutions without success:
- Run logstash 7.0
- Run logstash 7.4
- Upgrade beats client to 7.3.2
- Set file owner uid to 1000 (logstash)
- Set ssl_certificate_authorities to gd_bundle-g2-g1.crt (also tried with gdroot-g2.crt)
- curl -v --cacert … check fails to connect to 5044
I can provide any further information in a redacted form for security reasons.
Any suggestions how to approach this issue further?
