Elastic 2 node cluster: "Received fatal alert: certificate_unknown", but .p12 file with same certs and keys work

Hi

Elastic 6.8.0
Centos 7.6 (64-bit)

I have 2 node elastic cluster in my sandbox.

I created .p12 file myself with company certificates.

openssl pkcs12 -export -in company_domain.cer -inkey company_domain.key -name "elastic" -out mytest.p12

Then Imported trusted certificate (ca-pem.cer) via Keystore Explorer.

Copied mytest.p12 to both nodes.

Added into elasticsearch.yml file:

xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: full
xpack.security.transport.ssl.keystore.path: certs/mytest.p12
xpack.security.transport.ssl.truststore.path: certs/mytest.p12

Started elasticsearch Node1, all good, cluster in yellow state.
Started elasticsearch Node2, all good, cluster in green state.

Then

Deleted previous rows, and added these instead into elasticsearch.yml file (bear in mind that these are the exact same certificates and keys, that previously worked just fine):

Copied company_domain.key, company_domain.cer and ca-pem.cer to both nodes.

xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: full
xpack.security.transport.ssl.key: certs/company_domain.key
xpack.security.transport.ssl.certificate: certs/company_domain.cer
xpack.security.transport.ssl.certificate_authorities: certs/ca-pem.cer

Started elasticsearch Node1, all good, cluster in yellow state.
Started elasticsearch Node2, and I get:

javax.net.ssl.SSLException: Received fatal alert: certificate_unknown

(cluster stays in yellow state and nodes will not communicate with each other)

...
[2019-05-24T11:34:52,173][INFO ][o.e.n.Node               ] [dev-dc2-rk] initialized
[2019-05-24T11:34:52,173][INFO ][o.e.n.Node               ] [dev-dc2-rk] starting ...
[2019-05-24T11:34:52,280][INFO ][o.e.t.TransportService   ] [dev-dc2-rk] publish_address {10.61.2.18:9300}, bound_addresses {127.0.0.1:9300}, {10.61.2.18:9300}
[2019-05-24T11:34:52,311][INFO ][o.e.b.BootstrapChecks    ] [dev-dc2-rk] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2019-05-24T11:34:52,657][WARN ][o.e.t.OutboundHandler    ] [dev-dc2-rk] send message failed [channel: Netty4TcpChannel{localAddress=0.0.0.0/0.0.0.0:55656, remoteAddress=dev-dc1-rk.just.sise/10.51.2.18:9300}]
javax.net.ssl.SSLException: Received fatal alert: certificate_unknown
        at sun.security.ssl.Alerts.getSSLException(Alerts.java:208) ~[?:?]
        at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1647) ~[?:?]
        at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1615) ~[?:?]
        at sun.security.ssl.SSLEngineImpl.recvAlert(SSLEngineImpl.java:1781) ~[?:?]
        at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:1070) ~[?:?]
        at sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:896) ~[?:?]
        at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:766) ~[?:?]
        at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624) ~[?:1.8.0_212]
        at io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:295) ~[netty-handler-4.1.32.Final.jar:4.1.32.Final]
        at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1301) [netty-handler-4.1.32.Final.jar:4.1.32.Final]
        at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1203) [netty-handler-4.1.32.Final.jar:4.1.32.Final]
        at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1247) [netty-handler-4.1.32.Final.jar:4.1.32.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502) [netty-codec-4.1.32.Final.jar:4.1.32.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441) [netty-codec-4.1.32.Final.jar:4.1.32.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278) [netty-codec-4.1.32.Final.jar:4.1.32.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:656) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:556) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:510) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909) [netty-common-4.1.32.Final.jar:4.1.32.Final]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212]
[2019-05-24T11:34:52,666][WARN ][o.e.x.c.s.t.n.SecurityNetty4Transport] [dev-dc2-rk] client did not trust this server's certificate, closing connection Netty4TcpChannel{localAddress=0.0.0.0/0.0.0.0:55656, remoteAddress=dev-dc1-rk.just.sise/10.51.2.18:9300}
[2019-05-24T11:34:52,667][WARN ][o.e.x.c.s.t.n.SecurityNetty4Transport] [dev-dc2-rk] client did not trust this server's certificate, closing connection Netty4TcpChannel{localAddress=0.0.0.0/0.0.0.0:55656, remoteAddress=dev-dc1-rk.just.sise/10.51.2.18:9300}
[2019-05-24T11:34:53,423][WARN ][o.e.t.OutboundHandler    ] [dev-dc2-rk] send message failed [channel: Netty4TcpChannel{localAddress=0.0.0.0/0.0.0.0:55658, remoteAddress=dev-dc1-
...

This is very strange. I do not understand why these files won't work. Only .p12 seems to work.

Can you clarify where did you import that certificate? By the way , what is "Keystore Explorer" ?

Also, can we please get the logs from

dev-dc1-rk.just.sise/10.51.2.18

those will contain additional information on why the client ( 10.51.2.18 ) did not trust this server's ( 10.61.2.18 ) certificate.

Keystore Explorer, windows installer:
https://keystore-explorer.org/downloads.html

I can get these logs later on..

Raul

Ok, so I tried it again, and same result.

To make logs more readable, I kept these attempt logs separate:

  • attempt1 (keystore and truststore) - Result: cluster green
  • attempt2 (key, certificate, certificate_authorities) - Result: cluster in yellow state on both nodes

I used exact same elasticsearch.yml configuration values (and also files), as I mentioned on my first post.

But my problem, how can I upload log files here, it says only allowed are picture files.

Or should I just add whole log file text here..?

Ok, I post some of it into this post and the rest of it to the next post:
Logs are from dev-dc1-rk.just.sise/10.51.2.18, which is master node and attempt2

...
[2019-05-30T13:44:01,994][INFO ][o.e.c.r.a.AllocationService] [dev-dc1-rk] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[etoimik_failid][1], [etoimik_failid][0]] ...]).
[2019-05-30T13:44:06,066][WARN ][o.e.c.r.a.AllocationService] [dev-dc1-rk] [.watches][0] marking unavailable shards as stale: [Kxul4py1ScGDAAa-YnX1nA]
[2019-05-30T13:44:36,191][WARN ][o.e.t.TcpTransport       ] [dev-dc1-rk] exception caught on transport layer [Netty4TcpChannel{localAddress=0.0.0.0/0.0.0.0:9300, remoteAddress=/10.61.2.18:40998}], closing connection
io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:472) ~[netty-codec-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278) ~[netty-codec-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:656) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:556) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:510) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909) [netty-common-4.1.32.Final.jar:4.1.32.Final]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212]
Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
	at sun.security.ssl.Handshaker.checkThrown(Handshaker.java:1521) ~[?:?]
	at sun.security.ssl.SSLEngineImpl.checkTaskThrown(SSLEngineImpl.java:528) ~[?:?]
	at sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:802) ~[?:?]
	at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:766) ~[?:?]
	at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624) ~[?:1.8.0_212]
	at io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:295) ~[netty-handler-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1301) ~[netty-handler-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1203) ~[netty-handler-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1247) ~[netty-handler-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502) ~[netty-codec-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441) ~[netty-codec-4.1.32.Final.jar:4.1.32.Final]
	... 15 more
... to be continued..

Raul

Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
	at sun.security.ssl.Alerts.getSSLException(Alerts.java:192) ~[?:?]
	at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1709) ~[?:?]
	at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:318) ~[?:?]
	at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:310) ~[?:?]
	at sun.security.ssl.ServerHandshaker.clientCertificate(ServerHandshaker.java:1983) ~[?:?]
	at sun.security.ssl.ServerHandshaker.processMessage(ServerHandshaker.java:232) ~[?:?]
	at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1037) ~[?:?]
	at sun.security.ssl.Handshaker$1.run(Handshaker.java:970) ~[?:?]
	at sun.security.ssl.Handshaker$1.run(Handshaker.java:967) ~[?:?]
	at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_212]
	at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1459) ~[?:?]
	at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1464) ~[netty-handler-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1369) ~[netty-handler-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1203) ~[netty-handler-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1247) ~[netty-handler-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502) ~[netty-codec-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441) ~[netty-codec-4.1.32.Final.jar:4.1.32.Final]
	... 15 more
Caused by: sun.security.validator.ValidatorException: Extended key usage does not permit use for TLS client authentication
	at sun.security.validator.EndEntityChecker.checkTLSClient(EndEntityChecker.java:245) ~[?:?]
	at sun.security.validator.EndEntityChecker.check(EndEntityChecker.java:146) ~[?:?]
	at sun.security.validator.Validator.validate(Validator.java:274) ~[?:?]
	at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324) ~[?:?]
	at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:279) ~[?:?]
	at sun.security.ssl.X509TrustManagerImpl.checkClientTrusted(X509TrustManagerImpl.java:130) ~[?:?]
	at sun.security.ssl.ServerHandshaker.clientCertificate(ServerHandshaker.java:1970) ~[?:?]
	at sun.security.ssl.ServerHandshaker.processMessage(ServerHandshaker.java:232) ~[?:?]
	at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1037) ~[?:?]
	at sun.security.ssl.Handshaker$1.run(Handshaker.java:970) ~[?:?]
	at sun.security.ssl.Handshaker$1.run(Handshaker.java:967) ~[?:?]
	at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_212]
	at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1459) ~[?:?]
	at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1464) ~[netty-handler-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1369) ~[netty-handler-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1203) ~[netty-handler-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1247) ~[netty-handler-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502) ~[netty-codec-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441) ~[netty-codec-4.1.32.Final.jar:4.1.32.Final]
	... 15 more

The problem is with your company_domain.cer . When your CA signed it, it added the value 1.3.6.1.5.5.7.3.1( == The certificate can be used for Server Authentication only ) to the Extended Key Usage Field.

If you want to use a certificate instead of a keystore, then you'd have to regenerate the certificates.

Oh, yeah, you are absolutely correct, there is this:

Server Authentication (1.3.6.1.5.5.7.3.1)

But how keystore works fine with this same certificate..?

Regards
Raul

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.