Elastic 2 node cluster: "Received fatal alert: certificate_unknown", but .p12 file with same certs and keys work

raulk89 · May 24, 2019, 8:54am

Hi

Elastic 6.8.0
Centos 7.6 (64-bit)

I have 2 node elastic cluster in my sandbox.

I created .p12 file myself with company certificates.

openssl pkcs12 -export -in company_domain.cer -inkey company_domain.key -name "elastic" -out mytest.p12

Then Imported trusted certificate (ca-pem.cer) via Keystore Explorer.

Copied mytest.p12 to both nodes.

Added into elasticsearch.yml file:

xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: full
xpack.security.transport.ssl.keystore.path: certs/mytest.p12
xpack.security.transport.ssl.truststore.path: certs/mytest.p12

Started elasticsearch Node1, all good, cluster in yellow state.
Started elasticsearch Node2, all good, cluster in green state.

Then

Deleted previous rows, and added these instead into elasticsearch.yml file (bear in mind that these are the exact same certificates and keys, that previously worked just fine):

Copied company_domain.key, company_domain.cer and ca-pem.cer to both nodes.

xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: full
xpack.security.transport.ssl.key: certs/company_domain.key
xpack.security.transport.ssl.certificate: certs/company_domain.cer
xpack.security.transport.ssl.certificate_authorities: certs/ca-pem.cer

Started elasticsearch Node1, all good, cluster in yellow state.
Started elasticsearch Node2, and I get:

javax.net.ssl.SSLException: Received fatal alert: certificate_unknown

(cluster stays in yellow state and nodes will not communicate with each other)

...
[2019-05-24T11:34:52,173][INFO ][o.e.n.Node               ] [dev-dc2-rk] initialized
[2019-05-24T11:34:52,173][INFO ][o.e.n.Node               ] [dev-dc2-rk] starting ...
[2019-05-24T11:34:52,280][INFO ][o.e.t.TransportService   ] [dev-dc2-rk] publish_address {10.61.2.18:9300}, bound_addresses {127.0.0.1:9300}, {10.61.2.18:9300}
[2019-05-24T11:34:52,311][INFO ][o.e.b.BootstrapChecks    ] [dev-dc2-rk] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2019-05-24T11:34:52,657][WARN ][o.e.t.OutboundHandler    ] [dev-dc2-rk] send message failed [channel: Netty4TcpChannel{localAddress=0.0.0.0/0.0.0.0:55656, remoteAddress=dev-dc1-rk.just.sise/10.51.2.18:9300}]
javax.net.ssl.SSLException: Received fatal alert: certificate_unknown
        at sun.security.ssl.Alerts.getSSLException(Alerts.java:208) ~[?:?]
        at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1647) ~[?:?]
        at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1615) ~[?:?]
        at sun.security.ssl.SSLEngineImpl.recvAlert(SSLEngineImpl.java:1781) ~[?:?]
        at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:1070) ~[?:?]
        at sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:896) ~[?:?]
        at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:766) ~[?:?]
        at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624) ~[?:1.8.0_212]
        at io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:295) ~[netty-handler-4.1.32.Final.jar:4.1.32.Final]
        at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1301) [netty-handler-4.1.32.Final.jar:4.1.32.Final]
        at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1203) [netty-handler-4.1.32.Final.jar:4.1.32.Final]
        at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1247) [netty-handler-4.1.32.Final.jar:4.1.32.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502) [netty-codec-4.1.32.Final.jar:4.1.32.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441) [netty-codec-4.1.32.Final.jar:4.1.32.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278) [netty-codec-4.1.32.Final.jar:4.1.32.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:656) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:556) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:510) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909) [netty-common-4.1.32.Final.jar:4.1.32.Final]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212]
[2019-05-24T11:34:52,666][WARN ][o.e.x.c.s.t.n.SecurityNetty4Transport] [dev-dc2-rk] client did not trust this server's certificate, closing connection Netty4TcpChannel{localAddress=0.0.0.0/0.0.0.0:55656, remoteAddress=dev-dc1-rk.just.sise/10.51.2.18:9300}
[2019-05-24T11:34:52,667][WARN ][o.e.x.c.s.t.n.SecurityNetty4Transport] [dev-dc2-rk] client did not trust this server's certificate, closing connection Netty4TcpChannel{localAddress=0.0.0.0/0.0.0.0:55656, remoteAddress=dev-dc1-rk.just.sise/10.51.2.18:9300}
[2019-05-24T11:34:53,423][WARN ][o.e.t.OutboundHandler    ] [dev-dc2-rk] send message failed [channel: Netty4TcpChannel{localAddress=0.0.0.0/0.0.0.0:55658, remoteAddress=dev-dc1-
...

This is very strange. I do not understand why these files won't work. Only .p12 seems to work.

ikakavas · May 30, 2019, 7:14am

Can you clarify where did you import that certificate? By the way , what is "Keystore Explorer" ?

Also, can we please get the logs from

dev-dc1-rk.just.sise/10.51.2.18

those will contain additional information on why the client ( 10.51.2.18 ) did not trust this server's ( 10.61.2.18 ) certificate.

raulk89 · May 30, 2019, 7:21am

Keystore Explorer, windows installer:
https://keystore-explorer.org/downloads.html

I can get these logs later on..

Raul

raulk89 · May 30, 2019, 10:55am

Ok, so I tried it again, and same result.

To make logs more readable, I kept these attempt logs separate:

attempt1 (keystore and truststore) - Result: cluster green
attempt2 (key, certificate, certificate_authorities) - Result: cluster in yellow state on both nodes

I used exact same elasticsearch.yml configuration values (and also files), as I mentioned on my first post.

But my problem, how can I upload log files here, it says only allowed are picture files.

Or should I just add whole log file text here..?

Ok, I post some of it into this post and the rest of it to the next post:
Logs are from dev-dc1-rk.just.sise/10.51.2.18, which is master node and attempt2

...
[2019-05-30T13:44:01,994][INFO ][o.e.c.r.a.AllocationService] [dev-dc1-rk] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[etoimik_failid][1], [etoimik_failid][0]] ...]).
[2019-05-30T13:44:06,066][WARN ][o.e.c.r.a.AllocationService] [dev-dc1-rk] [.watches][0] marking unavailable shards as stale: [Kxul4py1ScGDAAa-YnX1nA]
[2019-05-30T13:44:36,191][WARN ][o.e.t.TcpTransport       ] [dev-dc1-rk] exception caught on transport layer [Netty4TcpChannel{localAddress=0.0.0.0/0.0.0.0:9300, remoteAddress=/10.61.2.18:40998}], closing connection
io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:472) ~[netty-codec-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278) ~[netty-codec-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:656) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:556) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:510) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470) [netty-transport-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909) [netty-common-4.1.32.Final.jar:4.1.32.Final]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212]
Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
	at sun.security.ssl.Handshaker.checkThrown(Handshaker.java:1521) ~[?:?]
	at sun.security.ssl.SSLEngineImpl.checkTaskThrown(SSLEngineImpl.java:528) ~[?:?]
	at sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:802) ~[?:?]
	at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:766) ~[?:?]
	at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624) ~[?:1.8.0_212]
	at io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:295) ~[netty-handler-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1301) ~[netty-handler-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1203) ~[netty-handler-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1247) ~[netty-handler-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502) ~[netty-codec-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441) ~[netty-codec-4.1.32.Final.jar:4.1.32.Final]
	... 15 more
... to be continued..

Raul

raulk89 · May 30, 2019, 11:06am

Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
	at sun.security.ssl.Alerts.getSSLException(Alerts.java:192) ~[?:?]
	at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1709) ~[?:?]
	at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:318) ~[?:?]
	at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:310) ~[?:?]
	at sun.security.ssl.ServerHandshaker.clientCertificate(ServerHandshaker.java:1983) ~[?:?]
	at sun.security.ssl.ServerHandshaker.processMessage(ServerHandshaker.java:232) ~[?:?]
	at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1037) ~[?:?]
	at sun.security.ssl.Handshaker$1.run(Handshaker.java:970) ~[?:?]
	at sun.security.ssl.Handshaker$1.run(Handshaker.java:967) ~[?:?]
	at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_212]
	at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1459) ~[?:?]
	at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1464) ~[netty-handler-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1369) ~[netty-handler-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1203) ~[netty-handler-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1247) ~[netty-handler-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502) ~[netty-codec-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441) ~[netty-codec-4.1.32.Final.jar:4.1.32.Final]
	... 15 more
Caused by: sun.security.validator.ValidatorException: Extended key usage does not permit use for TLS client authentication
	at sun.security.validator.EndEntityChecker.checkTLSClient(EndEntityChecker.java:245) ~[?:?]
	at sun.security.validator.EndEntityChecker.check(EndEntityChecker.java:146) ~[?:?]
	at sun.security.validator.Validator.validate(Validator.java:274) ~[?:?]
	at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324) ~[?:?]
	at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:279) ~[?:?]
	at sun.security.ssl.X509TrustManagerImpl.checkClientTrusted(X509TrustManagerImpl.java:130) ~[?:?]
	at sun.security.ssl.ServerHandshaker.clientCertificate(ServerHandshaker.java:1970) ~[?:?]
	at sun.security.ssl.ServerHandshaker.processMessage(ServerHandshaker.java:232) ~[?:?]
	at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1037) ~[?:?]
	at sun.security.ssl.Handshaker$1.run(Handshaker.java:970) ~[?:?]
	at sun.security.ssl.Handshaker$1.run(Handshaker.java:967) ~[?:?]
	at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_212]
	at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1459) ~[?:?]
	at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1464) ~[netty-handler-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1369) ~[netty-handler-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1203) ~[netty-handler-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1247) ~[netty-handler-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502) ~[netty-codec-4.1.32.Final.jar:4.1.32.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441) ~[netty-codec-4.1.32.Final.jar:4.1.32.Final]
	... 15 more

ikakavas · May 30, 2019, 11:17am

The problem is with your company_domain.cer . When your CA signed it, it added the value 1.3.6.1.5.5.7.3.1( == The certificate can be used for Server Authentication only ) to the Extended Key Usage Field.

If you want to use a certificate instead of a keystore, then you'd have to regenerate the certificates.

raulk89 · May 31, 2019, 9:59am

Oh, yeah, you are absolutely correct, there is this:

Server Authentication (1.3.6.1.5.5.7.3.1)

But how keystore works fine with this same certificate..?

Regards
Raul

system · June 28, 2019, 10:00am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
New Node Cert Issue Elasticsearch	2	555	May 24, 2018
Elasticsearch https secure communication Elasticsearch elastic-stack-security	4	531	March 11, 2020
SSL Communication Client and Cluster Error Elasticsearch	7	860	March 22, 2018
How configure .p12 certificate in Elasticsearch TLS/SSL? Elasticsearch elastic-stack-security	4	4564	November 26, 2018
Elasticsearch Cross Cluster Search Implementation Elasticsearch elastic-stack-security , ccs-cross-cluster-search	15	2122	January 28, 2021

Elastic 2 node cluster: "Received fatal alert: certificate_unknown", but .p12 file with same certs and keys work

Related topics