Empty certificate chain after upgrading from ES 7.8.1 to ES 7.9.2

Elastic Stack Elasticsearch
1

Hello,

After the upgrade from ES 7.8.1 to ES 7.9.2, Elastic threw in the logs an "empty certificate chain" exception.

I was using the following configuration:

xpack.security.transport.ssl.keystore.path: /usr/share/elasticsearch/config/certs/services.p12
xpack.security.transport.ssl.truststore.path: /usr/share/elasticsearch/config/certs/services.p12

The pkcs12 "services.p12" was created with, the service certificate and key and the root certificate

openssl pkcs12 -export -out services.p12 -in services.crt -inkey services.key  -name services -CAfile services-ca.crt -caname root -chain

I had to update the configuration to:

xpack.security.transport.ssl.keystore.path: /usr/share/elasticsearch/config/certs/services.p12
xpack.security.transport.ssl.truststore.path: /usr/share/elasticsearch/config/certs/services-ca.p12

Where "services-ca.p12" contains only CA root.

Was the previous configuration wrong?

1 Like
2

I'm seeing this same behavior change as well from 7.9.1 to 7.9.2.

3

Same here, any solution?

4

@ikakavas I saw your reply on some of the similar posts, maybe you have some insight.

I'll describe you what did I do.
I had a working 7.2.1 test cluster. I updated every node (ES, Kibana) to 7.9.2. ES started successfully but Kibana can't connect.

In the browser I got:

image
image1452×853 51.6 KB

Kibana logs (dropped "200" rows):

{"type":"log","@timestamp":"2020-10-13T10:47:47Z","tags":["warning","elasticsearch","security"],"pid":16769,"message":"Unable to revive connection: https://XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX:9200/"}
{"type":"log","@timestamp":"2020-10-13T10:47:47Z","tags":["warning","elasticsearch","security"],"pid":16769,"message":"No living connections"}
{"type":"log","@timestamp":"2020-10-13T10:47:47Z","tags":["error","http"],"pid":16769,"message":"{ Error: No Living connections\n at sendReqWithConnection (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:266:15)\n at next (/usr/share/kibana/node_modules/elasticsearch/src/lib/connection_pool.js:243:7)\n at process._tickCallback (internal/process/next_tick.js:61:11)\n message: 'No Living connections',\n body: undefined,\n status: undefined }"}
{"type":"error","@timestamp":"2020-10-13T10:47:47Z","tags":,"pid":16769,"level":"error","error":{"message":"Internal Server Error","name":"Error","stack":"Error: Internal Server Error\n
at HapiResponseAdapter.toInternalError (/usr/share/kibana/src/core/server/http/router/response_adapter.js:69:19)\n at Router.handle (/usr/share/kibana/src/core/server/http/router/router.js:170:34)\n at process._tickCallback (internal/process/next_tick.js:68:7)"},"url":{"protocol":null,"slashes":null,"auth":null,"host":null,"port":null,"hostname":null,"hash":null,"search":null,"query":{},"pathname":"/api/core/capabilities","path":"/api/core/capabilities","href":"/api/core/capabilities"},"message":"Internal Server Error"}
{"type":"response","@timestamp":"2020-10-13T10:47:47Z","tags":,"pid":16769,"method":"post","statusCode":500,"req":{"url":"/api/core/capabilities","method":"post","headers":{"host":"127.0.0.1:5601","connection":"close","content-length":"687","kbn-version":"7.9.2","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.121 Safari/537.36","content-type":"application/json","accept":"/","origin":"https://XXXXXXXX.com","sec-fetch-site":"same-origin","sec-fetch-mode":"cors","sec-fetch-dest":"empty","referer":"xxxxx.com, deflate, br","accept-language":"en-GB,en-US;q=0.9,en;q=0.8,hu;q=0.7"},"remoteAddress":"127.0.0.1","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.121 Safari/537.36","referer":"https://XXXXXXXX.com/login?next=%2F"},"res":{"statusCode":500,"responseTime":68,"contentLength":9},"message":"POST /api/core/capabilities 500 68ms - 9.0B"}

In ES, I have:

[WARN ][o.e.h.AbstractHttpServerTransport] [XXXXXXXXX] caught exception while handling client http traffic, closing connection Netty4HttpChannel{localAddress=/X.Y.Z.W:9200, remoteAddress=/X.Y.Z.W:46472}
io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: Empty server certificate chain
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:471) ~[netty-codec-4.1.49.Final.jar:4.1.49.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) ~[netty-codec-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [netty-common-4.1.49.Final.jar:4.1.49.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.49.Final.jar:4.1.49.Final]
at java.lang.Thread.run(Thread.java:832) [?:?]
Caused by: javax.net.ssl.SSLHandshakeException: Empty server certificate chain
at sun.security.ssl.Alert.createSSLException(Alert.java:131) ~[?:?]
at sun.security.ssl.Alert.createSSLException(Alert.java:117) ~[?:?]
at sun.security.ssl.TransportContext.fatal(TransportContext.java:356) ~[?:?]
at sun.security.ssl.TransportContext.fatal(TransportContext.java:312) ~[?:?]
at sun.security.ssl.TransportContext.fatal(TransportContext.java:303) ~[?:?]
at sun.security.ssl.CertificateMessage$T12CertificateConsumer.onCertificate(CertificateMessage.java:381) ~[?:?]
at sun.security.ssl.CertificateMessage$T12CertificateConsumer.consume(CertificateMessage.java:366) ~[?:?]
at sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:396) ~[?:?]
at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:480) ~[?:?]
at sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1267) ~[?:?]
at sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1254) ~[?:?]
at java.security.AccessController.doPrivileged(AccessController.java:691) ~[?:?]
at sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:1199) ~[?:?]
at io.netty.handler.ssl.SslHandler.runAllDelegatedTasks(SslHandler.java:1542) ~[netty-handler-4.1.49.Final.jar:4.1.49.Final]
at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1556) ~[netty-handler-4.1.49.Final.jar:4.1.49.Final]
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1440) ~[netty-handler-4.1.49.Final.jar:4.1.49.Final]
at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1267) ~[netty-handler-4.1.49.Final.jar:4.1.49.Final]
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1314) ~[netty-handler-4.1.49.Final.jar:4.1.49.Final]
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:501) ~[netty-codec-4.1.49.Final.jar:4.1.49.Final]
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:440) ~[netty-codec-4.1.49.Final.jar:4.1.49.Final]
... 16 more

Kibana.yml:

elasticsearch.requestTimeout: 120000
logging.dest: /var/log/kibana/kibana.log
pid.file: "/usr/share/kibana/pid.log"
server.ssl.certificate: "/path/to/kibana.cert.pem"
server.ssl.key: "/path/to/kibana.key.pem"
elasticsearch.hosts: ["https://XXXXXXXXXXXXXXXXXXXXXXX:9200"]
elasticsearch.username: "kibana_system"
elasticsearch.password: "XXXXXXXXXXXXXXXXXXXXXXX"
elasticsearch.ssl.certificateAuthorities: "/path/to/es-ca-chain.cert.pem"
elasticsearch.ssl.certificate: "/path/to/kibana.cert.pem"
elasticsearch.ssl.key: "/path/to/kibana.key.pem"
xpack.security.encryptionKey: "XXXXXXXXXXXXXXXXXXXXXXX"

elasticsearch.yml:

cluster.name: dev-cluster-elasticsearch
node.name: ${HOSTNAME}
path.logs: /var/log/elasticsearch
network.host: ["XXXXXXXXXXXXXXXX", "localhost"]
discovery.seed_hosts: ["XXXXXXXXXXXXXXXX"]
cluster.initial_master_nodes: ["XXXXXXXXXXXXXXXX"]

xpack.ml.enabled: false

xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.key: /path/to/elastic.key.pem
xpack.security.transport.ssl.certificate: /path/to/elastic.cert.pem
xpack.security.transport.ssl.certificate_authorities: ["/path/to/ca1.pem", "/path/to/ca2.pem"]

xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.verification_mode: certificate
xpack.security.http.ssl.client_authentication: required
xpack.security.http.ssl.key: /path/to/elastic.key.pem
xpack.security.http.ssl.certificate: /path/to/elastic.cert.pem
xpack.security.http.ssl.certificate_authorities: ["/path/to/ca1.pem", "/path/to/ca2.pem"]

The dev cluster was working fine before the upgrade. Any ideas?

5

See https://github.com/elastic/elasticsearch/issues/49510 and try changing xpack.security.http.ssl.client_authentication: required to xpack.security.http.ssl.client_authentication: optional .

Do you really ( as in on purpose ) want to do mTLS between kibana and elasticsearch ? Is there any specific reason for that ? The connection is protected via TLS eitherway and the client ( kibana ) is authenticating itself with a username and password already, there is no huge security gain by doing client TLS authentication too.

6

Thanks @ikakavas,
Hacking ES nodes is one of the new sports and I'd like to maximize security and narrow down who has access to the ES node. Though I'm not that well versed in SSL setups, so I usually go with the most secure one if possible. Also, I do have several other built-in and custom users connecting to ES. I don't see the point in changing the setting to "optional". I mean, the issue will likely go away, but from that point one could connect to ES with un+pw without providing a trusted CA, right?
Since this was working in previous versions, did something change specifically?
I'm sorry for these questions but I want to upgrade and keep the stack secured too.

7

Hi there,

You can keep client authentication to required ( as long as you don't enable a PKI realm in elasticsearch ) and have a functional setup by setting elasticsearch.ssl.alwaysPresentCertificate: true in your kibana.yml.

Hacking ES nodes is one of the new sports and I'd like to maximize security and narrow down who has access to the ES node. Though I'm not that well versed in SSL setups, so I usually go with the most secure one if possible

Sure thing. It all depends on your threat model. If you need/want to have additional authentication on the transport ( as in OSI not in Elasticsearch transport ) layer so that only clients that have a client certificate by a CA you trust can connect, that's the way to go.

How are these users authenticating to ES then ? I guess via a username and password since you have no PKI realm configured.

without providing a client certificate that is signed by a trusted CA, that is, but yes. Authentication will happen at the application layer ( elasticsearch ) , as it will happen with any of your users making requests to the REST APIs of elasticsearch.

I think I would expect this to not work even before 7.9.2 tbh, but I haven't checked recently

1 Like
8

Thank you @ikakavas!
As expected, the extra setting worked as a charm!

Yep, that's right, those will be tested a bit later.

Again, thank you for your thorough answer!

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.