# Empty certificate chain after upgrading from ES 7.8.1 to ES 7.9.2 **URL:** \ **Category:** Elasticsearch\ **Tags:** elastic-stack-security\ **Created:** [September 28, 2020, 8:19am UTC](https://discuss.elastic.co/t/empty-certificate-chain-after-upgrading-from-es-7-8-1-to-es-7-9-2/250145 "2020-09-28T08:19:57Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![Nicolas\_Labrot](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nicolas_labrot/32/2177_2.png) [@Nicolas\_Labrot](https://discuss.elastic.co/u/Nicolas_Labrot)\ **Post date:** [September 28, 2020, 8:19am UTC](https://discuss.elastic.co/t/empty-certificate-chain-after-upgrading-from-es-7-8-1-to-es-7-9-2/250145/1 "2020-09-28T08:19:58Z")
Hello, After the upgrade from ES 7.8.1 to ES 7.9.2, Elastic threw in the logs an "empty certificate chain" exception. I was using the following configuration: ``` xpack.security.transport.ssl.keystore.path: /usr/share/elasticsearch/config/certs/services.p12 xpack.security.transport.ssl.truststore.path: /usr/share/elasticsearch/config/certs/services.p12 ``` The pkcs12 "services.p12" was created with, the service certificate and key and the root certificate ``` openssl pkcs12 -export -out services.p12 -in services.crt -inkey services.key -name services -CAfile services-ca.crt -caname root -chain ``` I had to update the configuration to: ``` xpack.security.transport.ssl.keystore.path: /usr/share/elasticsearch/config/certs/services.p12 xpack.security.transport.ssl.truststore.path: /usr/share/elasticsearch/config/certs/services-ca.p12 ``` Where "services-ca.p12" contains only CA root. Was the previous configuration wrong? ---
**Author:** ![mschmit](https://avatars.discourse-cdn.com/v4/letter/m/eb9ed0/32.png) [@mschmit](https://discuss.elastic.co/u/mschmit)\ **Post date:** [September 29, 2020, 9:53pm UTC](https://discuss.elastic.co/t/empty-certificate-chain-after-upgrading-from-es-7-8-1-to-es-7-9-2/250145/2 "2020-09-29T21:53:34Z")
I'm seeing this same behavior change as well from 7.9.1 to 7.9.2. ---
**Author:** ![YvorL](https://avatars.discourse-cdn.com/v4/letter/y/9fc348/32.png) [@YvorL](https://discuss.elastic.co/u/YvorL)\ **Post date:** [October 13, 2020, 10:54am UTC](https://discuss.elastic.co/t/empty-certificate-chain-after-upgrading-from-es-7-8-1-to-es-7-9-2/250145/3 "2020-10-13T10:54:07Z")
Same here, any solution? ---
**Author:** ![YvorL](https://avatars.discourse-cdn.com/v4/letter/y/9fc348/32.png) [@YvorL](https://discuss.elastic.co/u/YvorL)\ **Post date:** [October 13, 2020, 1:36pm UTC](https://discuss.elastic.co/t/empty-certificate-chain-after-upgrading-from-es-7-8-1-to-es-7-9-2/250145/4 "2020-10-13T13:36:13Z")
@ikakavas I saw your reply on some of the similar posts, maybe you have some insight. I'll describe you what did I do. I had a working 7.2.1 test cluster. I updated every node (ES, Kibana) to 7.9.2. ES started successfully but Kibana can't connect. In the browser I got: ![image](https://us1.discourse-cdn.com/elastic/original/3X/f/6/f62d33d2b84e6cb304637f833598b5d2afcb5091.png) Kibana logs (dropped "200" rows): > {"type":"log","@timestamp":"2020-10-13T10:47:47Z","tags":["warning","elasticsearch","security"],"pid":16769,"message":"Unable to revive connection: [https://XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX:9200/](https://XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX:9200/)"} > {"type":"log","@timestamp":"2020-10-13T10:47:47Z","tags":["warning","elasticsearch","security"],"pid":16769,"message":"No living connections"} > {"type":"log","@timestamp":"2020-10-13T10:47:47Z","tags":["error","http"],"pid":16769,"message":"{ Error: No Living connections\n at sendReqWithConnection (/usr/share/kibana/node\_modules/elasticsearch/src/lib/transport.js:266:15)\n at next (/usr/share/kibana/node\_modules/elasticsearch/src/lib/connection\_pool.js:243:7)\n at process.\_tickCallback (internal/process/next\_tick.js:61:11)\n message: 'No Living connections',\n body: undefined,\n status: undefined }"} > {"type":"error","@timestamp":"2020-10-13T10:47:47Z","tags":,"pid":16769,"level":"error","error":{"message":"Internal Server Error","name":"Error","stack":"Error: Internal Server Error\n > at HapiResponseAdapter.toInternalError (/usr/share/kibana/src/core/server/http/router/response\_adapter.js:69:19)\n at Router.handle (/usr/share/kibana/src/core/server/http/router/router.js:170:34)\n at process.\_tickCallback (internal/process/next\_tick.js:68:7)"},"url":{"protocol":null,"slashes":null,"auth":null,"host":null,"port":null,"hostname":null,"hash":null,"search":null,"query":{},"pathname":"/api/core/capabilities","path":"/api/core/capabilities","href":"/api/core/capabilities"},"message":"Internal Server Error"} > {"type":"response","@timestamp":"2020-10-13T10:47:47Z","tags":,"pid":16769,"method":"post","statusCode":500,"req":{"url":"/api/core/capabilities","method":"post","headers":{"host":"127.0.0.1:5601","connection":"close","content-length":"687","kbn-version":"7.9.2","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.121 Safari/537.36","content-type":"application/json","accept":"_/_","origin":"[https://XXXXXXXX.com](https://XXXXXXXX.com)","sec-fetch-site":"same-origin","sec-fetch-mode":"cors","sec-fetch-dest":"empty","referer":"[xxxxx.com](https://XXXXXXX.com/login?next=%2F%22,%22accept-encoding%22:%22gzip), deflate, br","accept-language":"en-GB,en-US;q=0.9,en;q=0.8,hu;q=0.7"},"remoteAddress":"127.0.0.1","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.121 Safari/537.36","referer":"[https://XXXXXXXX.com/login?next=%2F"},"res":{"statusCode":500,"responseTime":68,"contentLength":9},"message":"POST](https://XXXXXXXX.com/login?next=%2F%22%7D,%22res%22:%7B%22statusCode%22:500,%22responseTime%22:68,%22contentLength%22:9%7D,%22message%22:%22POST) /api/core/capabilities 500 68ms - 9.0B"} In ES, I have: > [WARN][o.e.h.AbstractHttpServerTransport] [XXXXXXXXX] caught exception while handling client http traffic, closing connection Netty4HttpChannel{localAddress=/X.Y.Z.W:9200, remoteAddress=/X.Y.Z.W:46472} > io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: Empty server certificate chain > at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:471) ~[netty-codec-4.1.49.Final.jar:4.1.49.Final] > at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) ~[netty-codec-4.1.49.Final.jar:4.1.49.Final] > at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final] > at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final] > at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final] > at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.49.Final.jar:4.1.49.Final] > at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final] > at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final] > at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.49.Final.jar:4.1.49.Final] > at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.49.Final.jar:4.1.49.Final] > at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) [netty-transport-4.1.49.Final.jar:4.1.49.Final] > at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) [netty-transport-4.1.49.Final.jar:4.1.49.Final] > at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) [netty-transport-4.1.49.Final.jar:4.1.49.Final] > at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) [netty-transport-4.1.49.Final.jar:4.1.49.Final] > at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [netty-common-4.1.49.Final.jar:4.1.49.Final] > at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.49.Final.jar:4.1.49.Final] > at java.lang.Thread.run(Thread.java:832) [?:?] > Caused by: javax.net.ssl.SSLHandshakeException: Empty server certificate chain > at sun.security.ssl.Alert.createSSLException(Alert.java:131) ~[?:?] > at sun.security.ssl.Alert.createSSLException(Alert.java:117) ~[?:?] > at sun.security.ssl.TransportContext.fatal(TransportContext.java:356) ~[?:?] > at sun.security.ssl.TransportContext.fatal(TransportContext.java:312) ~[?:?] > at sun.security.ssl.TransportContext.fatal(TransportContext.java:303) ~[?:?] > at sun.security.ssl.CertificateMessage$T12CertificateConsumer.onCertificate(CertificateMessage.java:381) ~[?:?] > at sun.security.ssl.CertificateMessage$T12CertificateConsumer.consume(CertificateMessage.java:366) ~[?:?] > at sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:396) ~[?:?] > at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:480) ~[?:?] > at sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1267) ~[?:?] > at sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1254) ~[?:?] > at java.security.AccessController.doPrivileged(AccessController.java:691) ~[?:?] > at sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:1199) ~[?:?] > at io.netty.handler.ssl.SslHandler.runAllDelegatedTasks(SslHandler.java:1542) ~[netty-handler-4.1.49.Final.jar:4.1.49.Final] > at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1556) ~[netty-handler-4.1.49.Final.jar:4.1.49.Final] > at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1440) ~[netty-handler-4.1.49.Final.jar:4.1.49.Final] > at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1267) ~[netty-handler-4.1.49.Final.jar:4.1.49.Final] > at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1314) ~[netty-handler-4.1.49.Final.jar:4.1.49.Final] > at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:501) ~[netty-codec-4.1.49.Final.jar:4.1.49.Final] > at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:440) ~[netty-codec-4.1.49.Final.jar:4.1.49.Final] > ... 16 more Kibana.yml: > elasticsearch.requestTimeout: 120000 > logging.dest: /var/log/kibana/kibana.log > pid.file: "/usr/share/kibana/pid.log" > server.ssl.certificate: "/path/to/kibana.cert.pem" > server.ssl.key: "/path/to/kibana.key.pem" > elasticsearch.hosts: ["[https://XXXXXXXXXXXXXXXXXXXXXXX:9200](https://XXXXXXXXXXXXXXXXXXXXXXX:9200)"] > elasticsearch.username: "kibana\_system" > elasticsearch.password: "XXXXXXXXXXXXXXXXXXXXXXX" > elasticsearch.ssl.certificateAuthorities: "/path/to/es-ca-chain.cert.pem" > elasticsearch.ssl.certificate: "/path/to/kibana.cert.pem" > elasticsearch.ssl.key: "/path/to/kibana.key.pem" > xpack.security.encryptionKey: "XXXXXXXXXXXXXXXXXXXXXXX" elasticsearch.yml: > cluster.name: dev-cluster-elasticsearch > node.name: ${HOSTNAME} > path.logs: /var/log/elasticsearch > network.host: ["XXXXXXXXXXXXXXXX", "localhost"] > discovery.seed\_hosts: ["XXXXXXXXXXXXXXXX"] > cluster.initial\_master\_nodes: ["XXXXXXXXXXXXXXXX"] > > xpack.ml.enabled: false > > xpack.security.enabled: true > xpack.security.transport.ssl.enabled: true > xpack.security.transport.ssl.verification\_mode: certificate > xpack.security.transport.ssl.key: /path/to/elastic.key.pem > xpack.security.transport.ssl.certificate: /path/to/elastic.cert.pem > xpack.security.transport.ssl.certificate\_authorities: ["/path/to/ca1.pem", "/path/to/ca2.pem"] > > xpack.security.http.ssl.enabled: true > xpack.security.http.ssl.verification\_mode: certificate > xpack.security.http.ssl.client\_authentication: required > xpack.security.http.ssl.key: /path/to/elastic.key.pem > xpack.security.http.ssl.certificate: /path/to/elastic.cert.pem > xpack.security.http.ssl.certificate\_authorities: ["/path/to/ca1.pem", "/path/to/ca2.pem"] The dev cluster was working fine before the upgrade. Any ideas? ---
**Author:** ![ikakavas](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ikakavas/32/34430_2.png) [@ikakavas](https://discuss.elastic.co/u/ikakavas)\ **Post date:** [October 13, 2020, 2:14pm UTC](https://discuss.elastic.co/t/empty-certificate-chain-after-upgrading-from-es-7-8-1-to-es-7-9-2/250145/5 "2020-10-13T14:14:37Z")
See [https://github.com/elastic/elasticsearch/issues/49510](https://github.com/elastic/elasticsearch/issues/49510) and try changing `xpack.security.http.ssl.client_authentication: required` to `xpack.security.http.ssl.client_authentication: optional` . Do you really ( as in on purpose ) want to do mTLS between kibana and elasticsearch ? Is there any specific reason for that ? The connection is protected via TLS eitherway and the client ( kibana ) is authenticating itself with a username and password already, there is no huge security gain by doing client TLS authentication too. ---
**Author:** ![YvorL](https://avatars.discourse-cdn.com/v4/letter/y/9fc348/32.png) [@YvorL](https://discuss.elastic.co/u/YvorL)\ **Post date:** [October 13, 2020, 2:38pm UTC](https://discuss.elastic.co/t/empty-certificate-chain-after-upgrading-from-es-7-8-1-to-es-7-9-2/250145/6 "2020-10-13T14:38:28Z")
Thanks @ikakavas, Hacking ES nodes is one of the new sports and I'd like to maximize security and narrow down who has access to the ES node. Though I'm not that well versed in SSL setups, so I usually go with the most secure one if possible. Also, I do have several other built-in and custom users connecting to ES. I don't see the point in changing the setting to "optional". I mean, the issue will likely go away, but from that point one could connect to ES with un+pw without providing a trusted CA, right? Since this was working in previous versions, did something change specifically? I'm sorry for these questions but I want to upgrade and keep the stack secured too. ---
**Author:** ![ikakavas](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ikakavas/32/34430_2.png) [@ikakavas](https://discuss.elastic.co/u/ikakavas)\ **Post date:** [October 13, 2020, 7:53pm UTC](https://discuss.elastic.co/t/empty-certificate-chain-after-upgrading-from-es-7-8-1-to-es-7-9-2/250145/7 "2020-10-13T19:53:03Z")
Hi there, You can keep client authentication to `required` ( as long as you don't enable a PKI realm in elasticsearch ) and have a functional setup by setting `elasticsearch.ssl.alwaysPresentCertificate: true` in your kibana.yml. > Hacking ES nodes is one of the new sports and I'd like to maximize security and narrow down who has access to the ES node. Though I'm not that well versed in SSL setups, so I usually go with the most secure one if possible Sure thing. It all depends on your threat model. If you need/want to have additional authentication on the transport ( as in OSI not in Elasticsearch transport ) layer so that only clients that have a client certificate by a CA you trust can connect, that's the way to go. > [@YvorL](#): > > Also, I do have several other built-in and custom users connecting to ES. How are these users authenticating to ES then ? I guess via a username and password since you have no PKI realm configured. > [@YvorL](#): > > but from that point one could connect to ES with un+pw without providing a trusted CA, right? without providing a client certificate that is signed by a trusted CA, that is, but yes. Authentication will happen at the application layer ( elasticsearch ) , as it will happen with any of your users making requests to the REST APIs of elasticsearch. > [@YvorL](#): > > Since this was working in previous versions, did something change specifically? I think I would expect this to _not_ work even before 7.9.2 tbh, but I haven't checked recently ---
**Author:** ![YvorL](https://avatars.discourse-cdn.com/v4/letter/y/9fc348/32.png) [@YvorL](https://discuss.elastic.co/u/YvorL)\ **Post date:** [October 14, 2020, 8:50am UTC](https://discuss.elastic.co/t/empty-certificate-chain-after-upgrading-from-es-7-8-1-to-es-7-9-2/250145/8 "2020-10-14T08:50:43Z")
Thank you @ikakavas! As expected, the extra setting worked as a charm! > [@ikakavas](#): > > How are these users authenticating to ES then ? Yep, that's right, those will be tested a bit later. Again, thank you for your thorough answer! ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [November 11, 2020, 8:50am UTC](https://discuss.elastic.co/t/empty-certificate-chain-after-upgrading-from-es-7-8-1-to-es-7-9-2/250145/9 "2020-11-11T08:50:50Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.