# Elasticsearch \<-\> kibana mutual tls authentication: Empty client certificate chain **URL:** \ **Category:** Kibana\ **Tags:** elastic-stack-security\ **Created:** [May 30, 2022, 12:45pm UTC](https://discuss.elastic.co/t/elasticsearch-kibana-mutual-tls-authentication-empty-client-certificate-chain/305954 "2022-05-30T12:45:39Z")\ **Posts on this page:** 13\ **Page:** 1
**Author:** ![psylity](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/psylity/32/106331_2.png) [@psylity](https://discuss.elastic.co/u/psylity)\ **Post date:** [May 30, 2022, 12:45pm UTC](https://discuss.elastic.co/t/elasticsearch-kibana-mutual-tls-authentication-empty-client-certificate-chain/305954/1 "2022-05-30T12:45:39Z")
Hello! I'm using elasticsearch & kibana both 7.17.1 and can't run mutual tls authentication setup where both elasticsearch server and clients authenticate each other. I've followed [official documentation](https://www.elastic.co/guide/en/kibana/7.17/elasticsearch-mutual-tls.html) on this, and it contains this instruction: > 1. xpack.security.http.ssl.client\_authentication: "optional" Other possible option is "required". This means that elasticsearch will require client certificate. with "optional" setup everything works, but I believe there is **NO tls client auth** : ```auto curl --cacert ca.crt https://192.168.1.100:9200 { "name" : "elastic-1", "cluster_name" : "elasticsearch", "cluster_uuid" : "rgPoHJUdSv2TY8hk0dautQ", "version" : { "number" : "7.17.1", "build_flavor" : "default", "build_type" : "rpm", "build_hash" : "e5acb99f822233d62d6444ce45a4543dc1c8059a", "build_date" : "2022-02-23T22:20:54.153567231Z", "build_snapshot" : false, "lucene_version" : "8.11.1", "minimum_wire_compatibility_version" : "6.8.0", "minimum_index_compatibility_version" : "6.0.0-beta1" }, "tagline" : "You Know, for Search" } ``` As you see **I didn't provide client certificate to curl and still request is proceed.** If i change `xpack.security.http.ssl.client_authentication` to `required`, curl check works as expected: ```auto curl --cacert ca.crt https://192.168.1.100:9200 curl: (56) OpenSSL SSL_read: error:14094412:SSL routines:ssl3_read_bytes:sslv3 alert bad certificate, errno 0 curl --cacert ca.crt --cert kibana.crt --key kibana.key https://192.168.1.100:9200 { "name" : "elastic-1", "cluster_name" : "elasticsearch", ... ``` but kibana fails to connect to the elasticsearch instance. I get this `ConnectionError: socket hang up - Local: 192.168.1.101:35278, Remote: 192.168.1.100:9200 ` in kibana log and `javax.net.ssl.SSLHandshakeException: Empty client certificate chain` in elasticsearch log. This exception also occurs if i run curl without specifying the client certificate. I make a conclusion that **kibana doesn't provide client certificate** to the elasticsearch server. my elasticsearch.yml: ```auto xpack.security.transport.ssl.enabled: true xpack.security.http.ssl.enabled: true xpack.security.http.ssl.certificate: elasticsearch.crt xpack.security.http.ssl.key: elasticsearch.key xpack.security.http.ssl.certificate_authorities: ca.crt xpack.security.http.ssl.client_authentication: "required" # xpack.security.http.ssl.client_authentication: "optional" ``` kibana.yml: ```auto elasticsearch.hosts: ["https://192.168.1.100:9200"] elasticsearch.ssl.verificationMode: certificate elasticsearch.ssl.certificate: /etc/kibana/kibana.crt elasticsearch.ssl.key: /etc/kibana/kibana.key elasticsearch.ssl.certificateAuthorities: ["/etc/kibana/ca.crt"] ``` What is the right way to solve this issue? ---
**Author:** ![ikakavas](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ikakavas/32/34430_2.png) [@ikakavas](https://discuss.elastic.co/u/ikakavas)\ **Post date:** [May 30, 2022, 2:19pm UTC](https://discuss.elastic.co/t/elasticsearch-kibana-mutual-tls-authentication-empty-client-certificate-chain/305954/3 "2022-05-30T14:19:03Z")
Have you done step 5 in the docs that you reference ? ( Set up a PKI realm in Elasticsearch ? ) ---
**Author:** ![psylity](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/psylity/32/106331_2.png) [@psylity](https://discuss.elastic.co/u/psylity)\ **Post date:** [May 30, 2022, 2:48pm UTC](https://discuss.elastic.co/t/elasticsearch-kibana-mutual-tls-authentication-empty-client-certificate-chain/305954/4 "2022-05-30T14:48:18Z")
Sure, I did: ```auto xpack.security.authc.realms.pki.realm1.order: 1 xpack.security.authc.realms.pki.realm1.certificate_authorities: "/etc/elasticsearch/ca.crt" xpack.security.authc.realms.native.realm2.order: 2 ``` ---
**Author:** ![Justin\_Cranford](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/justin_cranford/32/85302_2.png) [@Justin\_Cranford](https://discuss.elastic.co/u/Justin_Cranford)\ **Post date:** [May 30, 2022, 5:01pm UTC](https://discuss.elastic.co/t/elasticsearch-kibana-mutual-tls-authentication-empty-client-certificate-chain/305954/5 "2022-05-30T17:01:33Z")
It does sound as if Kibana is not sending a TLS client certificate chain. Or perhaps it is a another configuration issue which triggers the same symptom but for a different cause. Question, does it help if you change the format of these setting to be a list? - xpack.security.http.ssl.certificate\_authorities - xpack.security.authc.realms.pki.realm1.certificate\_authorities They would have a similar array format as this Kibana setting. - Elasticsearch.ssl.certificateAuthorities: ["/etc/kibana/ca.crt"] Question, have to tried enabling debug in Elasticsearch? If you have a working curl connection using mTLS, you should be able to call this Elasticsearch API to enable transient trace logging for Elasticsearch security. If the issue is happening at the application layer for PKI realm, you may get more useful diagnostic logging. ```auto PUT /_cluster/settings { "transient": { "logger.org.elasticsearch.xpack.security": "trace" } } ``` If the TLS client issue is at the TLS layer, you can try adding JVM option to Elasticsearch startup to log TLS debug from the JVM. ---
**Author:** ![psylity](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/psylity/32/106331_2.png) [@psylity](https://discuss.elastic.co/u/psylity)\ **Post date:** [May 30, 2022, 5:17pm UTC](https://discuss.elastic.co/t/elasticsearch-kibana-mutual-tls-authentication-empty-client-certificate-chain/305954/6 "2022-05-30T17:17:05Z")
Thank you for your answer! Reformatting this settings to lists changed nothing. As for the `trace`: ```auto curl --cacert ca.crt --cert kibana.crt --key kibana.key -X PUT -H "Content-Type: application/json" -d @enable-trace.json https://192.168.1.100:9200/_cluster/settings {"acknowledged":true,"persistent":{},"transient":{"logger":{"org":{"elasticsearch":{"xpack":{"security":"trace"}}}}}} ``` I get a tons of this messages: ```auto tail -f /var/log/elasticsearch/elasticsearch.log <...> [2022-05-30T17:11:01,720][DEBUG][o.e.x.s.a.a.OptOutQueryCache] [hostname] [.kibana_7.17.1_001] not opting out of the query cache; authorization is not allowed [2022-05-30T17:11:01,720][DEBUG][o.e.x.s.a.a.OptOutQueryCache] [hostname] [.kibana_7.17.1_001] not opting out of the query cache; authorization is not allowed [2022-05-30T17:11:01,720][DEBUG][o.e.x.s.a.a.OptOutQueryCache] [hostname] [.kibana_7.17.1_001] not opting out of the query cache; authorization is not allowed [2022-05-30T17:11:01,720][DEBUG][o.e.x.s.a.a.OptOutQueryCache] [hostname] [.kibana_7.17.1_001] not opting out of the query cache; authorization is not allowed [2022-05-30T17:11:04,623][DEBUG][o.e.x.s.a.a.OptOutQueryCache] [hostname] [.kibana_task_manager_7.17.1_001] not opting out of the query cache; authorization is not allowed [2022-05-30T17:11:04,623][DEBUG][o.e.x.s.a.a.OptOutQueryCache] [hostname] [.kibana_task_manager_7.17.1_001] not opting out of the query cache; authorization is not allowed [2022-05-30T17:11:04,623][DEBUG][o.e.x.s.a.a.OptOutQueryCache] [hostname] [.kibana_task_manager_7.17.1_001] not opting out of the query cache; authorization is not allowed [2022-05-30T17:11:04,623][DEBUG][o.e.x.s.a.a.OptOutQueryCache] [hostname] [.kibana_task_manager_7.17.1_001] not opting out of the query cache; authorization is not allowed <...> ``` but there is no new info around the exception: ```auto <...> Caused by: javax.net.ssl.SSLHandshakeException: Empty client certificate chain <...> ``` Maybe there is a special switch to debug TLS connection on kibana side? ---
**Author:** ![Justin\_Cranford](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/justin_cranford/32/85302_2.png) [@Justin\_Cranford](https://discuss.elastic.co/u/Justin_Cranford)\ **Post date:** [May 30, 2022, 6:29pm UTC](https://discuss.elastic.co/t/elasticsearch-kibana-mutual-tls-authentication-empty-client-certificate-chain/305954/7 "2022-05-30T18:29:00Z")
That feedback is helpful. If there are no new application logs for PKI, then it probablly means the TLS connection is stopping during the TLS hankshake. If you add **-Djavax.net.debug=all** to a file in Elasticsearch/config/jvm.options, it will output verbose TLS settings during startup and runtime. When connecting from Kibana to Elasticsearch, look for the ClientHello request received by Elasticsearch port 9200 TLS listener, to see if the TLS client cert chain is being received from Kibana. Even if it is received, there might be a validation error causing Elasticsearch to reject the chain. ---
**Author:** ![Justin\_Cranford](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/justin_cranford/32/85302_2.png) [@Justin\_Cranford](https://discuss.elastic.co/u/Justin_Cranford)\ **Post date:** [May 30, 2022, 6:30pm UTC](https://discuss.elastic.co/t/elasticsearch-kibana-mutual-tls-authentication-empty-client-certificate-chain/305954/8 "2022-05-30T18:30:23Z")
Correction. You can add **-Djavax.net.debug=all** in Elasticsearch/config/jvm.options file, or put it in a text file you add under the Elasticsearch/config/jvm.options.d/ directory. ---
**Author:** ![Justin\_Cranford](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/justin_cranford/32/85302_2.png) [@Justin\_Cranford](https://discuss.elastic.co/u/Justin_Cranford)\ **Post date:** [May 31, 2022, 12:58pm UTC](https://discuss.elastic.co/t/elasticsearch-kibana-mutual-tls-authentication-empty-client-certificate-chain/305954/9 "2022-05-31T12:58:50Z")
Something else that might be worth trying is `xpack.security.http.ssl.verification_mode=certificate`. I noticed this setting is not listed in the docs, but it does get applied and defaults to full. An issue has been opened to add it to the docs. In other words, it looks like you set Transport TLS verification\_mode to certificate, so try setting HTTP TLS verification\_mode to certificate. It is not clear if that is the issue, but debug logs from **-Djavax.net.debug=all** should tell you if an issue is happening at the TLS handshake level. Another one that is useful in some situations is **-Djava.security.debug=certpath**. That specifically helps debug validation of certificate chains, including validation of cert extensions. ---
**Author:** ![TimV](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/timv/32/13162_2.png) [@TimV](https://discuss.elastic.co/u/TimV)\ **Post date:** [June 2, 2022, 2:01am UTC](https://discuss.elastic.co/t/elasticsearch-kibana-mutual-tls-authentication-empty-client-certificate-chain/305954/10 "2022-06-02T02:01:30Z")
> [@psylity](#): > > I make a conclusion that **kibana doesn't provide client certificate** to the Elasticsearch server. This is by design. See the [`elasticsearch.ssl.alwaysPresentCertificate`](https://www.elastic.co/guide/en/kibana/current/settings.html#elasticsearch-ssl-alwaysPresentCertificate) setting. You cannot safely - require client authentication - _and_ use those certs for PKI authentication - _and_ have a working Kibana install Kibana need to be able to make connections to Elasticsearch as both itself and as the logged in user. If you require Kibana to present a certificate _and_ you use that to authenticate the user for which the connection is intended, then Kibana no longer has a way to authentication as the logged in user - it will always authenticate as itself. You have to give up one of those 3 properties. - You can have PKI authentication, plus Kibana, but you need to allow Kibana to connect without a certificate when perform request on behalf of users - You can require certificates, but then you cannot use them for PKI authentication (on connections coming from Kibana) - Or you can not use Kibana. There are some complicated setups that can be used to make this work, but they involve having multiple issuing CAs and then use only a subset of the CAs for PKI, and the others are only used for network control. ---
**Author:** ![psylity](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/psylity/32/106331_2.png) [@psylity](https://discuss.elastic.co/u/psylity)\ **Post date:** [June 2, 2022, 7:38am UTC](https://discuss.elastic.co/t/elasticsearch-kibana-mutual-tls-authentication-empty-client-certificate-chain/305954/11 "2022-06-02T07:38:40Z")
Thank you! Came out that I misunderstood the concept. Despite of the concept of CN \<-\> role mapping my idea was that there are different levels of authentication - one is transport layer (tls) and other - basic, which is running on top of tls. In other words mutual TLS is needed to authenticate the client (kibana) and the service (Elasticsearch) and the upper layer auth is http basic - to authenticate users. As for `elasticsearch.ssl.alwaysPresentCertificate` - am I right that to make it work I need: - to disable client auth; - to specify a client certificate with full Elasticsearch access in kibana config. ---
**Author:** ![TimV](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/timv/32/13162_2.png) [@TimV](https://discuss.elastic.co/u/TimV)\ **Post date:** [June 2, 2022, 8:45am UTC](https://discuss.elastic.co/t/elasticsearch-kibana-mutual-tls-authentication-empty-client-certificate-chain/305954/12 "2022-06-02T08:45:36Z")
From what you've written, I assume you don't want to use PKI authentication. Within the Elastic Stack we use the following terminology: - **SSL Client Authentication** (aka _Mutual TLS_, aka _Client Certificates_) means enabling (either required or optional) client side certificates on a TLS connection, typically the HTTP port. - **PKI** means using client certificates to determine the authenticated _User_ in Elasticsearch. If you just want Mutual TLS, then you want to enable `ssl.client_authentication` but you don't want to enable the PKI realm. If that is the case, then you can: - In Elasticsearch, set `xpack.security.http.ssl.client_authentication: required` so that every client requires a certificate. - In Kibana set `elasticsearch.ssl.alwaysPresentCertificate: true` so that Kibana always provides a certificate to Elasticsearch - Do not configure a PKI realm in Elasticsearch (remove it if you have one) That will mean that every Elasticsearch client (including Kibana) will be required to present a trusted certificate, _and also_ present some other form of authentication (such as a username + password, or API Key). ---
**Author:** ![Lokesh\_Sachdeva](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lokesh_sachdeva/32/105782_2.png) [@Lokesh\_Sachdeva](https://discuss.elastic.co/u/Lokesh_Sachdeva)\ **Post date:** [June 5, 2022, 9:04pm UTC](https://discuss.elastic.co/t/elasticsearch-kibana-mutual-tls-authentication-empty-client-certificate-chain/305954/13 "2022-06-05T21:04:29Z")
HI..I have a same issues for authenticating Kibana with ElasicSearch. Client Authentication is working fine with parameter: xpack.security.http.ssl.client\_authentication: "optional". With configuration as "Required", its gives exception/error as "Empty Client Certificate Chain" Error. Kindly let me know, what else configuration changes you have performed in your docker file for Client Authentication. Please find below my docker-compose.yml parameters: Kibana.yml ```auto Environment: - SERVERNAME=kibana - ELASTICSEARCH_HOSTS=https://es01:9200 - ELASTICSEARCH_USERNAME=kibana_system - ELASTICSEARCH_PASSWORD=${KIBANA_PASSWORD} - ELASTICSEARCH_SSL_CERTIFICATEAUTHORITIES=["config/certs/ca/ca.crt"] - SERVER_SSL_ENABLED=true - SERVER_SSL_CERTIFICATE=config/certs/Kibana_Server/kibana_server.crt - SERVER_SSL_KEY=config/certs/Kibana_Server/kibana_server.key - ELASTICSEARCH_SSL_CERTIFICATE=config/certs/Kibana_Server/kibana_server.crt - ELASTICSEARCH_SSL_KEY=config/certs/Kibana_Server/kibana_server.key - XPACK_SECURITY_SAMESITECOOKIES=None - XPACK_SECURITY_ENABLED=true - ELASTICSEARCH_SSL_VERIFICATION_MODE=certificate - ELASTICSEARCH_SSL_ALWAYSPRESENTCERTIFICATE=true ``` Elasticsearch.yml ```auto environment: - xpack.security.enabled=true - xpack.security.http.ssl.enabled=true - xpack.security.http.ssl.key=certs/es01/es01.key - xpack.security.http.ssl.certificate=certs/es01/es01.crt - xpack.security.http.ssl.certificate_authorities=certs/ca/ca.crt - xpack.security.http.ssl.verification_mode=certificate - xpack.security.transport.ssl.enabled=false - xpack.security.transport.ssl.key=certs/es01/es01.key - xpack.security.transport.ssl.certificate=certs/es01/es01.crt - xpack.security.transport.ssl.certificate_authorities=certs/ca/ca.crt - xpack.security.transport.ssl.verification_mode=certificate - xpack.license.self_generated.type=${LICENSE} - discovery.type=single-node - xpack.security.http.ssl.client_authentication=required ``` ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 3, 2022, 9:05pm UTC](https://discuss.elastic.co/t/elasticsearch-kibana-mutual-tls-authentication-empty-client-certificate-chain/305954/14 "2022-07-03T21:05:27Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.