Elasticsearch <-> kibana mutual tls authentication: Empty client certificate chain

psylity · May 30, 2022, 12:45pm

Hello! I'm using elasticsearch & kibana both 7.17.1 and can't run mutual tls authentication setup where both elasticsearch server and clients authenticate each other.
I've followed official documentation on this, and it contains this instruction:

xpack.security.http.ssl.client_authentication: "optional"

Other possible option is "required". This means that elasticsearch will require client certificate.

with "optional" setup everything works, but I believe there is NO tls client auth:

curl --cacert ca.crt  https://192.168.1.100:9200
{
  "name" : "elastic-1",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "rgPoHJUdSv2TY8hk0dautQ",
  "version" : {
    "number" : "7.17.1",
    "build_flavor" : "default",
    "build_type" : "rpm",
    "build_hash" : "e5acb99f822233d62d6444ce45a4543dc1c8059a",
    "build_date" : "2022-02-23T22:20:54.153567231Z",
    "build_snapshot" : false,
    "lucene_version" : "8.11.1",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

As you see I didn't provide client certificate to curl and still request is proceed.

If i change xpack.security.http.ssl.client_authentication to required, curl check works as expected:

curl --cacert ca.crt https://192.168.1.100:9200
curl: (56) OpenSSL SSL_read: error:14094412:SSL routines:ssl3_read_bytes:sslv3 alert bad certificate, errno 0

curl --cacert ca.crt --cert kibana.crt --key kibana.key https://192.168.1.100:9200
{
  "name" : "elastic-1",
  "cluster_name" : "elasticsearch",
...

but kibana fails to connect to the elasticsearch instance. I get this
ConnectionError: socket hang up - Local: 192.168.1.101:35278, Remote: 192.168.1.100:9200 in kibana log and javax.net.ssl.SSLHandshakeException: Empty client certificate chain in elasticsearch log. This exception also occurs if i run curl without specifying the client certificate.

I make a conclusion that kibana doesn't provide client certificate to the elasticsearch server.

my elasticsearch.yml:

xpack.security.transport.ssl.enabled: true
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.certificate: elasticsearch.crt
xpack.security.http.ssl.key: elasticsearch.key
xpack.security.http.ssl.certificate_authorities: ca.crt
xpack.security.http.ssl.client_authentication: "required"
# xpack.security.http.ssl.client_authentication: "optional"

kibana.yml:

elasticsearch.hosts: ["https://192.168.1.100:9200"]
elasticsearch.ssl.verificationMode: certificate
elasticsearch.ssl.certificate: /etc/kibana/kibana.crt 
elasticsearch.ssl.key: /etc/kibana/kibana.key 
elasticsearch.ssl.certificateAuthorities: [ "/etc/kibana/ca.crt" ]

What is the right way to solve this issue?

ikakavas · May 30, 2022, 2:19pm

Have you done step 5 in the docs that you reference ? ( Set up a PKI realm in Elasticsearch ? )

psylity · May 30, 2022, 2:48pm

Sure, I did:

xpack.security.authc.realms.pki.realm1.order: 1
xpack.security.authc.realms.pki.realm1.certificate_authorities: "/etc/elasticsearch/ca.crt"
xpack.security.authc.realms.native.realm2.order: 2

Justin_Cranford · May 30, 2022, 5:01pm

It does sound as if Kibana is not sending a TLS client certificate chain. Or perhaps it is a another configuration issue which triggers the same symptom but for a different cause.

Question, does it help if you change the format of these setting to be a list?

xpack.security.http.ssl.certificate_authorities
xpack.security.authc.realms.pki.realm1.certificate_authorities

They would have a similar array format as this Kibana setting.

Elasticsearch.ssl.certificateAuthorities: [ "/etc/kibana/ca.crt" ]

Question, have to tried enabling debug in Elasticsearch? If you have a working curl connection using mTLS, you should be able to call this Elasticsearch API to enable transient trace logging for Elasticsearch security. If the issue is happening at the application layer for PKI realm, you may get more useful diagnostic logging.

PUT /_cluster/settings
{
  "transient": {
    "logger.org.elasticsearch.xpack.security": "trace"
  }
}

If the TLS client issue is at the TLS layer, you can try adding JVM option to Elasticsearch startup to log TLS debug from the JVM.

psylity · May 30, 2022, 5:17pm

Thank you for your answer!
Reformatting this settings to lists changed nothing.

As for the trace:

curl --cacert ca.crt --cert kibana.crt --key kibana.key -X PUT -H "Content-Type: application/json" -d @enable-trace.json https://192.168.1.100:9200/_cluster/settings
{"acknowledged":true,"persistent":{},"transient":{"logger":{"org":{"elasticsearch":{"xpack":{"security":"trace"}}}}}}

I get a tons of this messages:

tail -f /var/log/elasticsearch/elasticsearch.log
<...>
[2022-05-30T17:11:01,720][DEBUG][o.e.x.s.a.a.OptOutQueryCache] [hostname] [.kibana_7.17.1_001] not opting out of the query cache; authorization is not allowed
[2022-05-30T17:11:01,720][DEBUG][o.e.x.s.a.a.OptOutQueryCache] [hostname] [.kibana_7.17.1_001] not opting out of the query cache; authorization is not allowed
[2022-05-30T17:11:01,720][DEBUG][o.e.x.s.a.a.OptOutQueryCache] [hostname] [.kibana_7.17.1_001] not opting out of the query cache; authorization is not allowed
[2022-05-30T17:11:01,720][DEBUG][o.e.x.s.a.a.OptOutQueryCache] [hostname] [.kibana_7.17.1_001] not opting out of the query cache; authorization is not allowed
[2022-05-30T17:11:04,623][DEBUG][o.e.x.s.a.a.OptOutQueryCache] [hostname] [.kibana_task_manager_7.17.1_001] not opting out of the query cache; authorization is not allowed
[2022-05-30T17:11:04,623][DEBUG][o.e.x.s.a.a.OptOutQueryCache] [hostname] [.kibana_task_manager_7.17.1_001] not opting out of the query cache; authorization is not allowed
[2022-05-30T17:11:04,623][DEBUG][o.e.x.s.a.a.OptOutQueryCache] [hostname] [.kibana_task_manager_7.17.1_001] not opting out of the query cache; authorization is not allowed
[2022-05-30T17:11:04,623][DEBUG][o.e.x.s.a.a.OptOutQueryCache] [hostname] [.kibana_task_manager_7.17.1_001] not opting out of the query cache; authorization is not allowed
<...>

but there is no new info around the exception:

<...>
Caused by: javax.net.ssl.SSLHandshakeException: Empty client certificate chain
<...>

Maybe there is a special switch to debug TLS connection on kibana side?

Justin_Cranford · May 30, 2022, 6:29pm

That feedback is helpful. If there are no new application logs for PKI, then it probablly means the TLS connection is stopping during the TLS hankshake. If you add -Djavax.net.debug=all to a file in Elasticsearch/config/jvm.options, it will output verbose TLS settings during startup and runtime. When connecting from Kibana to Elasticsearch, look for the ClientHello request received by Elasticsearch port 9200 TLS listener, to see if the TLS client cert chain is being received from Kibana. Even if it is received, there might be a validation error causing Elasticsearch to reject the chain.

Justin_Cranford · May 30, 2022, 6:30pm

Correction. You can add -Djavax.net.debug=all in Elasticsearch/config/jvm.options file, or put it in a text file you add under the Elasticsearch/config/jvm.options.d/ directory.

Justin_Cranford · May 31, 2022, 12:58pm

Something else that might be worth trying is xpack.security.http.ssl.verification_mode=certificate. I noticed this setting is not listed in the docs, but it does get applied and defaults to full. An issue has been opened to add it to the docs.

In other words, it looks like you set Transport TLS verification_mode to certificate, so try setting HTTP TLS verification_mode to certificate. It is not clear if that is the issue, but debug logs from -Djavax.net.debug=all should tell you if an issue is happening at the TLS handshake level.

Another one that is useful in some situations is -Djava.security.debug=certpath. That specifically helps debug validation of certificate chains, including validation of cert extensions.

TimV · June 2, 2022, 2:01am

This is by design.
See the elasticsearch.ssl.alwaysPresentCertificate setting.

You cannot safely

require client authentication
and use those certs for PKI authentication
and have a working Kibana install

Kibana need to be able to make connections to Elasticsearch as both itself and as the logged in user.

If you require Kibana to present a certificate and you use that to authenticate the user for which the connection is intended, then Kibana no longer has a way to authentication as the logged in user - it will always authenticate as itself.

You have to give up one of those 3 properties.

You can have PKI authentication, plus Kibana, but you need to allow Kibana to connect without a certificate when perform request on behalf of users
You can require certificates, but then you cannot use them for PKI authentication (on connections coming from Kibana)
Or you can not use Kibana.

There are some complicated setups that can be used to make this work, but they involve having multiple issuing CAs and then use only a subset of the CAs for PKI, and the others are only used for network control.

psylity · June 2, 2022, 7:38am

Thank you! Came out that I misunderstood the concept.

Despite of the concept of CN <-> role mapping my idea was that there are different levels of authentication - one is transport layer (tls) and other - basic, which is running on top of tls.
In other words mutual TLS is needed to authenticate the client (kibana) and the service (Elasticsearch) and the upper layer auth is http basic - to authenticate users.

As for elasticsearch.ssl.alwaysPresentCertificate - am I right that to make it work I need:

to disable client auth;
to specify a client certificate with full Elasticsearch access in kibana config.

TimV · June 2, 2022, 8:45am

From what you've written, I assume you don't want to use PKI authentication.

Within the Elastic Stack we use the following terminology:

SSL Client Authentication (aka Mutual TLS, aka Client Certificates) means enabling (either required or optional) client side certificates on a TLS connection, typically the HTTP port.
PKI means using client certificates to determine the authenticated User in Elasticsearch.

If you just want Mutual TLS, then you want to enable ssl.client_authentication but you don't want to enable the PKI realm.

If that is the case, then you can:

In Elasticsearch, set xpack.security.http.ssl.client_authentication: required so that every client requires a certificate.
In Kibana set elasticsearch.ssl.alwaysPresentCertificate: true so that Kibana always provides a certificate to Elasticsearch
Do not configure a PKI realm in Elasticsearch (remove it if you have one)

That will mean that every Elasticsearch client (including Kibana) will be required to present a trusted certificate, and also present some other form of authentication (such as a username + password, or API Key).

Lokesh_Sachdeva · June 5, 2022, 9:04pm

HI..I have a same issues for authenticating Kibana with ElasicSearch. Client Authentication is working fine with parameter: xpack.security.http.ssl.client_authentication: "optional".

With configuration as "Required", its gives exception/error as "Empty Client Certificate Chain" Error.

Kindly let me know, what else configuration changes you have performed in your docker file for Client Authentication.

Please find below my docker-compose.yml parameters:

Kibana.yml

Environment:
      - SERVERNAME=kibana
      - ELASTICSEARCH_HOSTS=https://es01:9200
      - ELASTICSEARCH_USERNAME=kibana_system
      - ELASTICSEARCH_PASSWORD=${KIBANA_PASSWORD}
      - ELASTICSEARCH_SSL_CERTIFICATEAUTHORITIES=["config/certs/ca/ca.crt"]
      - SERVER_SSL_ENABLED=true
      - SERVER_SSL_CERTIFICATE=config/certs/Kibana_Server/kibana_server.crt
      - SERVER_SSL_KEY=config/certs/Kibana_Server/kibana_server.key
      - ELASTICSEARCH_SSL_CERTIFICATE=config/certs/Kibana_Server/kibana_server.crt
      - ELASTICSEARCH_SSL_KEY=config/certs/Kibana_Server/kibana_server.key
      - XPACK_SECURITY_SAMESITECOOKIES=None
      - XPACK_SECURITY_ENABLED=true
      - ELASTICSEARCH_SSL_VERIFICATION_MODE=certificate
      - ELASTICSEARCH_SSL_ALWAYSPRESENTCERTIFICATE=true

Elasticsearch.yml

environment:
      - xpack.security.enabled=true
      - xpack.security.http.ssl.enabled=true
      - xpack.security.http.ssl.key=certs/es01/es01.key
      - xpack.security.http.ssl.certificate=certs/es01/es01.crt
      - xpack.security.http.ssl.certificate_authorities=certs/ca/ca.crt
      - xpack.security.http.ssl.verification_mode=certificate
      - xpack.security.transport.ssl.enabled=false
      - xpack.security.transport.ssl.key=certs/es01/es01.key
      - xpack.security.transport.ssl.certificate=certs/es01/es01.crt
      - xpack.security.transport.ssl.certificate_authorities=certs/ca/ca.crt
      - xpack.security.transport.ssl.verification_mode=certificate
      - xpack.license.self_generated.type=${LICENSE}
      - discovery.type=single-node
      - xpack.security.http.ssl.client_authentication=required

system · July 3, 2022, 9:05pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Mutual TLS authentication between Kibana and Elasticsearch keeping client_authentication "required" Kibana elastic-stack-security	3	289	February 3, 2023
Mutual tls/ssl on elasticsearch Elastic Security elastic-stack-security	27	4645	November 4, 2022
Mutual tls/ssl on elasticsearch Elasticsearch elastic-stack-security	25	7101	October 8, 2019
Which certificate should I prepare when enabling mutual tls authentication? Elasticsearch	4	3268	May 11, 2020
Conf Kibana and shield Elasticsearch elastic-stack-security	6	1314	December 15, 2016

Elasticsearch <-> kibana mutual tls authentication: Empty client certificate chain

Related topics