Logstash fails to start input.beats after elk 6.8 to 7.3 upgrade

After 6.8.0 to 7.3.2 upgrade (no issues in 7.0 upgrade assistant in kibana) Elastic fails to start input.beats with the error below:

[2019-10-01T14:30:32,879][WARN ][org.logstash.beats.Server] Exception caught in channel initializer
javax.net.ssl.SSLException: failed to set certificate and key
        at io.netty.handler.ssl.ReferenceCountedOpenSslServerContext.newSessionContext(ReferenceCountedOpenSslServerContext.java:140) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
        at io.netty.handler.ssl.OpenSslServerContext.<init>(OpenSslServerContext.java:352) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
        at io.netty.handler.ssl.OpenSslServerContext.<init>(OpenSslServerContext.java:335) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
        at io.netty.handler.ssl.SslContext.newServerContextInternal(SslContext.java:422) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
        at io.netty.handler.ssl.SslContextBuilder.build(SslContextBuilder.java:447) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
        at org.logstash.netty.SslSimpleBuilder.build(SslSimpleBuilder.java:128) ~[logstash-input-beats-6.0.1.jar:?]
        at org.logstash.beats.Server$BeatsInitializer.initChannel(Server.java:131) ~[logstash-input-beats-6.0.1.jar:?]
        at org.logstash.beats.Server$BeatsInitializer.initChannel(Server.java:101) [logstash-input-beats-6.0.1.jar:?]
        at io.netty.channel.ChannelInitializer.initChannel(ChannelInitializer.java:115) [netty-all-4.1.30.Final.jar:4.1.30.Final]
        at io.netty.channel.ChannelInitializer.handlerAdded(ChannelInitializer.java:107) [netty-all-4.1.30.Final.jar:4.1.30.Final]
        at io.netty.channel.DefaultChannelPipeline.callHandlerAdded0(DefaultChannelPipeline.java:637) [netty-all-4.1.30.Final.jar:4.1.30.Final]
        at io.netty.channel.DefaultChannelPipeline.access$000(DefaultChannelPipeline.java:46) [netty-all-4.1.30.Final.jar:4.1.30.Final]
        at io.netty.channel.DefaultChannelPipeline$PendingHandlerAddedTask.execute(DefaultChannelPipeline.java:1487) [netty-all-4.1.30.Final.jar:4.1.30.Final]
        at io.netty.channel.DefaultChannelPipeline.callHandlerAddedForAllHandlers(DefaultChannelPipeline.java:1161) [netty-all-4.1.30.Final.jar:4.1.30.Final]
        at io.netty.channel.DefaultChannelPipeline.invokeHandlerAddedIfNeeded(DefaultChannelPipeline.java:686) [netty-all-4.1.30.Final.jar:4.1.30.Final]
        at io.netty.channel.AbstractChannel$AbstractUnsafe.register0(AbstractChannel.java:510) [netty-all-4.1.30.Final.jar:4.1.30.Final]
        at io.netty.channel.AbstractChannel$AbstractUnsafe.access$200(AbstractChannel.java:423) [netty-all-4.1.30.Final.jar:4.1.30.Final]
        at io.netty.channel.AbstractChannel$AbstractUnsafe$1.run(AbstractChannel.java:482) [netty-all-4.1.30.Final.jar:4.1.30.Final]
        at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) [netty-all-4.1.30.Final.jar:4.1.30.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404) [netty-all-4.1.30.Final.jar:4.1.30.Final]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:462) [netty-all-4.1.30.Final.jar:4.1.30.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:897) [netty-all-4.1.30.Final.jar:4.1.30.Final]
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-all-4.1.30.Final.jar:4.1.30.Final]
        at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: java.security.KeyStoreException: Key protection  algorithm not found: java.security.KeyStoreException: Certificate chain is not valid
        at sun.security.pkcs12.PKCS12KeyStore.setKeyEntry(PKCS12KeyStore.java:704) ~[?:?]
        at sun.security.pkcs12.PKCS12KeyStore.engineSetKeyEntry(PKCS12KeyStore.java:601) ~[?:?]
        at sun.security.util.KeyStoreDelegator.engineSetKeyEntry(KeyStoreDelegator.java:111) ~[?:?]
        at java.security.KeyStore.setKeyEntry(KeyStore.java:1174) ~[?:?]
        at io.netty.handler.ssl.SslContext.buildKeyStore(SslContext.java:1004) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
        at io.netty.handler.ssl.ReferenceCountedOpenSslServerContext.newSessionContext(ReferenceCountedOpenSslServerContext.java:126) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
        ... 23 more
Caused by: java.security.KeyStoreException: Certificate chain is not valid
        at sun.security.pkcs12.PKCS12KeyStore.setKeyEntry(PKCS12KeyStore.java:651) ~[?:?]
        at sun.security.pkcs12.PKCS12KeyStore.engineSetKeyEntry(PKCS12KeyStore.java:601) ~[?:?]
        at sun.security.util.KeyStoreDelegator.engineSetKeyEntry(KeyStoreDelegator.java:111) ~[?:?]

My logstash setings for beats input are

beats {
port => 5044
ssl => true
ssl_certificate => "/logstash/logs.redacted.com.pem"
ssl_key => "/logstash/logs.redacted.com.key"
type => "beats"
}

I'm using x502 ssl certificate signed by GoDaddy
X509v3 Basic Constraints: critical
CA:FALSE
X509v3 Extended Key Usage:
TLS Web Server Authentication, TLS Web Client Authentication

The pem and key files set was working with previous versions incl. 6.8.0 and beats endpoint was receiving data. Elk is running under docker swarm, with configuration and cert files mounted using bind.

I wasn't able to find relevant information on this issue so tried several solutions without success:

  • Run logstash 7.0
  • Run logstash 7.4
  • Upgrade beats client to 7.3.2
  • Set file owner uid to 1000 (logstash)
  • Set ssl_certificate_authorities to gd_bundle-g2-g1.crt (also tried with gdroot-g2.crt)
  • curl -v --cacert … check fails to connect to 5044

I can provide any further information in a redacted form for security reasons.

Any suggestions how to approach this issue further?

Ok it is fixed now.

After some research it turned out that the certificate chain was not ordered which apparently is validated as part of the new JDK Logstash docker image is shipped with.

If somebody has similar issue, the steps for me was:

  • Get certificates list with Owner/Issuer data --> keytool -printcert -v -file the-x502-pem-file
  • Order the certs in the file so Issuer[i] == Owner[i+1].

More details in validateChain method here https://hg.openjdk.java.net/jdk/jdk11/file/e1e60f75cd39/src/java.base/share/classes/sun/security/pkcs12/PKCS12KeyStore.java#l1526

This obviously was Java related but it would be nice to have a hint in the elastic docs somewhere visible :smiley:

Cheers!

1 Like

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.