Can Elastic work with Windows certificates?

lachezar.uzunov · February 11, 2021, 8:27am

Hello everyone.

I created elastic stack of 3 nodes and it was working pretty fine until I have to replace the elastic generated certificate with one from windows server CA. I do not seem to find information of rather it is possible or not. I am keep getting error while trying to start elastic with the windows cert, such as:

[2021-02-11T03:13:58,194][WARN ][o.e.t.TcpTransport       ] [es-master] exception caught on transport layer [Netty4TcpChannel{localAddress=/127.0.0.1:35218, remoteAddress=/127.0.0.1:9300}], closing connection                                                              
io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: Received fatal alert: bad_certificate
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:471) ~[netty-codec-4.1.49.Final.jar:4.1.49.Final]                                                                                                                                 
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) ~[netty-codec-4.1.49.Final.jar:4.1.49.Final]                                                                                                                                
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]                                                                                                           
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]                                                                                                           
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]                                                                                                             
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.49.Final.jar:4.1.49.Final]                                                                                                                  
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]                                                                                                           
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]                                                                                                           
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.49.Final.jar:4.1.49.Final]                                                                                                                           
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.49.Final.jar:4.1.49.Final]                                                                                                                    
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) [netty-transport-4.1.49.Final.jar:4.1.49.Final]                                                                                                                                  
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [netty-common-4.1.49.Final.jar:4.1.49.Final]                                                                                                                          
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.49.Final.jar:4.1.49.Final]
        at java.lang.Thread.run(Thread.java:832) [?:?]
Caused by: javax.net.ssl.SSLHandshakeException: Received fatal alert: bad_certificate
        at sun.security.ssl.Alert.createSSLException(Alert.java:131) ~[?:?]
        at sun.security.ssl.Alert.createSSLException(Alert.java:117) ~[?:?]
        at sun.security.ssl.TransportContext.fatal(TransportContext.java:356) ~[?:?]
        at sun.security.ssl.Alert$AlertConsumer.consume(Alert.java:293) ~[?:?]
        at sun.security.ssl.TransportContext.dispatch(TransportContext.java:202) ~[?:?]
        at sun.security.ssl.SSLTransport.decode(SSLTransport.java:171) ~[?:?]
        at sun.security.ssl.SSLEngineImpl.decode(SSLEngineImpl.java:736) ~[?:?]
        at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:691) ~[?:?]
        at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:506) ~[?:?]
        at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:482) ~[?:?]
        at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:637) ~[?:?]
        at io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:282) ~[netty-handler-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1372) ~[netty-handler-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1267) ~[netty-handler-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1314) ~[netty-handler-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:501) ~[netty-codec-4.1.49.Final.jar:4.1.49.Final]                                                                                                             
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:440) ~[netty-codec-4.1.49.Final.jar:4.1.49.Final]                                                                                                                                 
        ... 16 more

And also that:

[2021-02-11T03:38:42,714][WARN ][o.e.t.TcpTransport       ] [es-master] exception caught on transport layer [Netty4TcpChannel{localAddress=/[0:0:0:0:0:0:0:1]:9300, remoteAddress=/[0:0:0:0:0:0:0:1]:46496}], closing connection                                              
io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: Empty client certificate chain
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:471) ~[netty-codec-4.1.49.Final.jar:4.1.49.Final]                                                                                                                                 
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) ~[netty-codec-4.1.49.Final.jar:4.1.49.Final]                                                                                                                                
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]                                                                                                           
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]                                                                                                           
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]                                                                                                             
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.49.Final.jar:4.1.49.Final]                                                                                                                  
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]                                                                                                           
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]                                                                                                           
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.49.Final.jar:4.1.49.Final]                                                                                                                           
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.49.Final.jar:4.1.49.Final]                                                                                                                    
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) [netty-transport-4.1.49.Final.jar:4.1.49.Final]                                                                                                                                  
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [netty-common-4.1.49.Final.jar:4.1.49.Final]                                                                                                                          
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.49.Final.jar:4.1.49.Final]
        at java.lang.Thread.run(Thread.java:832) [?:?]
Caused by: javax.net.ssl.SSLHandshakeException: Empty client certificate chain
        at sun.security.ssl.Alert.createSSLException(Alert.java:131) ~[?:?]
        at sun.security.ssl.Alert.createSSLException(Alert.java:117) ~[?:?]
        at sun.security.ssl.TransportContext.fatal(TransportContext.java:356) ~[?:?]
        at sun.security.ssl.TransportContext.fatal(TransportContext.java:312) ~[?:?]
        at sun.security.ssl.TransportContext.fatal(TransportContext.java:303) ~[?:?]
        at

When navigating to https://ip:9200 I am presented with the standard login screen, but when enter credentials the system does not let me in and just refreshes the screen.

Do anyone have idea is that is possible?

TimV · February 15, 2021, 12:57am

This looks like you have misconfigured one or more of your ES nodes.

It is exceptionally difficult diagnose configuration problems without seeing your elasticsearch.yml file.

lachezar.uzunov · February 15, 2021, 7:04am

Sure, here it it:

# ---------------------------------- Cluster -----------------------------------
cluster.name: elastic
# ------------------------------------ Node ------------------------------------
node.name: "es-master"
# ------------------------------------ Paths ------------------------------------
path.data: /data/data
path.logs: /data/logs
# ----------------------------------- Memory -----------------------------------
# ---------------------------------- Network -----------------------------------
network.host: 0.0.0.0
http.port: 9200
# --------------------------------- Discovery ----------------------------------
discovery.zen.minimum_master_nodes: 2
discovery.zen.ping.unicast.hosts:  ["172.16.249.44", "172.16.249.45", "172.16.249.46"]
cluster.initial_master_nodes: ["172.16.249.44",  "172.16.249.45", "172.16.249.46"]
# --------------------------------- Security -----------------------------------
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true 
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: /etc/elasticsearch/elastic_strong.pfx
xpack.security.transport.ssl.truststore.path: /etc/elasticsearch/elastic_strong.pfx
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.keystore.path: "/etc/elasticsearch/elastic_strong.pfx"
xpack.security.http.ssl.truststore.path: /etc/elasticsearch/elastic_strong.pfx

They use the same config.

lachezar.uzunov · February 16, 2021, 12:26pm

The problem is that the elasticearch does not read and work with your global linux key/trust store. The error represents that the chain cannot be validated (trusted). Which means it cannot read cert chain and respectively validate it. Cert I was trying to work with is from windows CA. My machines are all Debian 10. When using certs from windows CA it cannot read the whole chain by using only .pfx or .p12 cert. Then I tried generating CSR request to my organisation and it worked by removing keystore and trustore and replacing them with key, cert and ca. CA is the key here since it will force elastic to read the chain.

Here is my working config:

# ---------------------------------- Cluster -----------------------------------
cluster.name: elastic
# ------------------------------------ Node ------------------------------------
node.name: "es-master"
# ------------------------------------ Paths ------------------------------------
path.data: /data/data
path.logs: /data/logs
# ----------------------------------- Memory -----------------------------------
# ---------------------------------- Network -----------------------------------
network.host: 0.0.0.0
http.port: 9200
# --------------------------------- Discovery ----------------------------------
discovery.zen.minimum_master_nodes: 2
discovery.zen.ping.unicast.hosts:  ["IP1", "IP2", "IP3"]
cluster.initial_master_nodes: ["IP1","IP2","IP3"]
# --------------------------------- Security -----------------------------------
xpack.security.enabled: true

xpack.security.transport.ssl.verification_mode: certificate

xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.certificate: "http-elastic.cer"
xpack.security.http.ssl.key: "http-elastic.key"

xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.certificate: "http-elastic.cer"
xpack.security.transport.ssl.key: "http-elastic.key"

xpack.security.http.ssl.certificate_authorities: [ "CA-ROOT.DER.cer", "CA-SUB.DER.cer" ]
xpack.security.transport.ssl.certificate_authorities: [ "CA-ROOT.DER.cer", "CA-SUB.DER.cer" ]

Keep in mind that your cert must work as a server and a client simultaneously. For that case ask your CA to create cert which is containing "Digital Signature" and "Key Encipherment (a0)" in "Key Usage" variable.

Peace.

TimV · February 17, 2021, 2:32am

It is likely that this was your problem.
If you were issued with that file by your CA, then it is almost certainly that case that it is a valid keystore, but not a valid truststore.

system · March 17, 2021, 2:33am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.