Unable to connect to kibana with a secure cluster

Hello everyone,
I have configured a cluster of 5 secure nodes with this configuration (adapted on each node :slight_smile:

xpack.security.http.ssl:
  enabled: true
  keystore.path: /etc/elasticsearch/new/http.p12
  truststore.path: /etc/elasticsearch/new/http.p12
  client_authentication: required
  verification_mode: full
# Enable encryption and mutual authentication between cluster node
xpack.security.transport.ssl:
  enabled: true
  verification_mode: full
  client_authentication: required
  keystore.path: /etc/elasticsearch/new/node-1.p12
  truststore.path: /etc/elasticsearch/new/node-1.p12

For the http part I generated one certificate per cluster node (5) with the command :
./bin/elasticsearch-certutil http

But I can't connect to kibana

server.port: 5601
server.host: "192.168.1.80"
server.publicBaseUrl: "https://192.168.1.80:5601"
server.name: "kibana1"

server.ssl.enabled: true
#Generated with bin/elasticsearch-certutil cert and the same CA as between the elasticsearch cluster nodes
server.ssl.keystore.path: "/etc/kibana/certs/kibana1.p12"
server.ssl.truststore.path: "/etc/kibana/certs/kibana1.p12"
server.ssl.clientAuthentication: required

elasticsearch.hosts: ["https://192.168.1.10:9200", "https://192.168.1.20:9200","https://192.168.1.30:9200", "https://192.168.1.40:9200", "https://192.168.1.50:9200"]
#Generated with bin/elasticsearch-certutil http
elasticsearch.ssl.certificateAuthorities: [ "/etc/kibana/certs/elasticsearch-ca.pem" ]

If someone can help me because I can't find the solution.

Best regards,

Why, what happens when you try?

Thanks for your answer.
When I try connect to kibana with my browser nothing happens.
Port 5601 is open.

Is there a command to test if the connection between kibana and elasticsearch is done correctly?
What is the curl command to test the cluster status in ssl?

now my cluster does not form after reboot..
I have this error on my master nodes (2 master and 1 voting_only)

org.elasticsearch.transport.NodeNotConnectedException: [node-2][192.168.1.20:9300] Node not connected
        at org.elasticsearch.transport.ClusterConnectionManager.getConnection(ClusterConnectionManager.java:280) ~[elasticsearch-8.4.1.jar:?]
        at org.elasticsearch.transport.TransportService.getConnection(TransportService.java:808) ~[elasticsearch-8.4.1.jar:?]
        at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:718) ~[elasticsearch-8.4.1.jar:?]
        at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.start(TransportNodesAction.java:243) ~[elasticsearch-8.4.1.jar:?]
        at org.elasticsearch.action.support.nodes.TransportNodesAction.doExecute(TransportNodesAction.java:122) ~[elasticsearch-8.4.1.jar:?]
        at org.elasticsearch.action.support.nodes.TransportNodesAction.doExecute(TransportNodesAction.java:39) ~[elasticsearch-8.4.1.jar:?]
        at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:86) ~[elasticsearch-8.4.1.jar:?]
        at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$3(SecurityActionFilter.java:163) ~[?:?]
        at org.elasticsearch.action.ActionListener$DelegatingFailureActionListener.onResponse(ActionListener.java:245) ~[elasticsearch-8.4.1.jar:?]
        at org.elasticsearch.xpack.security.authz.AuthorizationService.authorizeSystemUser(AuthorizationService.java:620) ~[?:?]
        at org.elasticsearch.xpack.security.authz.AuthorizationService.authorize(AuthorizationService.java:257) ~[?:?]
        at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$4(SecurityActionFilter.java:159) ~[?:?]
        at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:162) ~[elasticsearch-8.4.1.jar:?]
        at org.elasticsearch.action.ActionListener$MappedActionListener.onResponse(ActionListener.java:127) ~[elasticsearch-8.4.1.jar:?]
        at org.elasticsearch.xpack.security.authc.AuthenticatorChain.authenticateAsync(AuthenticatorChain.java:93) ~[?:?]
        at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:171) ~[?:?]
        at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.applyInternal(SecurityActionFilter.java:155) ~[?:?]
        at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$apply$0(SecurityActionFilter.java:104) ~[?:?]
        at org.elasticsearch.xpack.core.security.SecurityContext.executeAsInternalUser(SecurityContext.java:121) ~[?:?]
        at org.elasticsearch.xpack.core.security.SecurityContext.executeAsSystemUser(SecurityContext.java:130) ~[?:?]
        at org.elasticsearch.xpack.core.security.SecurityContext.executeAsSystemUser(SecurityContext.java:126) ~[?:?]
        at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.apply(SecurityActionFilter.java:104) ~[?:?]
        at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:84) ~[elasticsearch-8.4.1.jar:?]
        at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:61) ~[elasticsearch-8.4.1.jar:?]
        at org.elasticsearch.tasks.TaskManager.registerAndExecute(TaskManager.java:186) ~[elasticsearch-8.4.1.jar:?]
        at org.elasticsearch.client.internal.node.NodeClient.executeLocally(NodeClient.java:112) ~[elasticsearch-8.4.1.jar:?]
        at org.elasticsearch.client.internal.node.NodeClient.doExecute(NodeClient.java:90) ~[elasticsearch-8.4.1.jar:?]
        at org.elasticsearch.client.internal.support.AbstractClient.execute(AbstractClient.java:380) ~[elasticsearch-8.4.1.jar:?]
        at org.elasticsearch.client.internal.support.AbstractClient$ClusterAdmin.execute(AbstractClient.java:676) ~[elasticsearch-8.4.1.jar:?]
        at org.elasticsearch.client.internal.support.AbstractClient$ClusterAdmin.nodesStats(AbstractClient.java:774) ~[elasticsearch-8.4.1.jar:?]
        at org.elasticsearch.cluster.InternalClusterInfoService$AsyncRefresh.execute(InternalClusterInfoService.java:184) ~[elasticsearch-8.4.1.jar:?]
        at org.elasticsearch.cluster.InternalClusterInfoService.refreshAsync(InternalClusterInfoService.java:409) ~[elasticsearch-8.4.1.jar:?]
        at org.elasticsearch.cluster.InternalClusterInfoService.clusterChanged(InternalClusterInfoService.java:152) ~[elasticsearch-8.4.1.jar:?]
        at org.elasticsearch.cluster.service.ClusterApplierService.callClusterStateListener(ClusterApplierService.java:558) ~[elasticsearch-8.4.1.jar:?]
        at org.elasticsearch.cluster.service.ClusterApplierService.callClusterStateListeners(ClusterApplierService.java:544) ~[elasticsearch-8.4.1.jar:?]
        at org.elasticsearch.cluster.service.ClusterApplierService.applyChanges(ClusterApplierService.java:504) ~[elasticsearch-8.4.1.jar:?]
        at org.elasticsearch.cluster.service.ClusterApplierService.runTask(ClusterApplierService.java:428) ~[elasticsearch-8.4.1.jar:?]
        at org.elasticsearch.cluster.service.ClusterApplierService$UpdateTask.run(ClusterApplierService.java:154) ~[elasticsearch-8.4.1.jar:?]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:710) ~[elasticsearch-8.4.1.jar:?]
        at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:257) ~[elasticsearch-8.4.1.jar:?]
        at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:223) ~[elasticsearch-8.4.1.jar:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
        at java.lang.Thread.run(Thread.java:833) ~[?:?]

Best regards,

About the errors on the cluster, some nodes were locked in a single cluster because I had not removed the option :
Configure cluster.initial_master_nodes
after the formation of the cluster.

I had to reform the cluster with this procedure:

  • Shut down all the nodes.
  • Delete each node completely by deleting the content of their data folders. (path.data in elasticsearch.ym)
  • Configure cluster.initial_master_nodes as described above. (to be deleted after formation of the cluster ! !!!!!)
  • Configure discovery.seed_hosts or discovery.seed_providers and other relevant - discovery settings.
  • Restart all nodes and verify that they have formed a single cluster.

Hello @warkolm,
For Kibana when I test the port 5601, with telnet 192.168.1.80 5601 the connection is refused .
The port is well opened in the firewall.

Best regards,

First of all, you need make sure your Elasticsearch cluster is correctly formed and up running. You can test that with curl to check the Cluster health API. Or at least you should not see any errors in the server logs.

Second, your kibana configuration is likely a mismatch with ES configuration. You configured ES to require client_authentication on HTTP:

xpack.security.http.ssl:
  enabled: true
  keystore.path: /etc/elasticsearch/new/http.p12
  truststore.path: /etc/elasticsearch/new/http.p12
  client_authentication: required

But you did not configure Kibana to send its certificate. I guess you probably misunderstood the following kibana settings

server.ssl.enabled: true
#Generated with bin/elasticsearch-certutil cert and the same CA as between the elasticsearch cluster nodes
server.ssl.keystore.path: "/etc/kibana/certs/kibana1.p12"
server.ssl.truststore.path: "/etc/kibana/certs/kibana1.p12"
server.ssl.clientAuthentication: required

The above settings mandates client_authentication when a client talks to Kibana, not when Kbiana talks to Elasticsearch. For that you need to use elasticsearch.ssl.keystore.path and its friends. Please refer to the Kibana's settings doc for details.

Thank you so much for your answer @Yang_Wang
I need to keep these settings ;

server.ssl.enabled: true
---
#Generated with bin/elasticsearch-certutil cert and the same CA as between the elasticsearch cluster nodes
server.ssl.keystore.path: "/etc/kibana/certs/kibana1.p12"
server.ssl.truststore.path: "/etc/kibana/certs/kibana1.p12"
server.ssl.clientAuthentication: required

should I add these parameters?

elasticsearch.ssl.certificateAuthorities: [ "/etc/kibana/certs/elasticsearch-ca.pem" ]
elasticsearch.ssl.keystore.path: "/etc/kibana/certs/http.p12"
elasticsearch.ssl.alwaysPresentCertificate: true.
elasticsearch.ssl.verificationMode: full

In elasticsearch.ssl.keystore.path I have to add a certificate generated with the certutil http command and fill in a kibana node? or is it the same as in server.ssl.keystore.path?

I don't understand because when generating http certificates an elasticsearch-ca.pem is generated to trust the chain.

Thank you for your precious help

With the configuration in my previous post, I get this error message:

journalctl -u kibana -f
FATAL Error: [config validation of [elasticsearch].ssl.keystore]: could not parse object value from json input

That usually means syntax error in the configuration file. Please double check that. A few things I noticed:

  1. You may need configure elasticsearch.ssl.keystore.password if the keystore is password protected or empty string if it is not (depending on how it is generated)
  2. There is a period after elasticsearch.ssl.alwaysPresentCertificate: true in your shareed configuration.

I managed to correct the previous mistakes.
Kibana starts correctly but it tells me :
Unable to retrieve version information from Elasticsearch nodes

And when I connect with my browser, I get this message after bypassing the certificate warning:
SSL_ERROR_RX_CERTIFICATE_REQUIRED_ALERT

Thanks again for your help @Yang_Wang

The previous errors were related to the keystore password which I solved with the following commands:

bin/kibana-keystore add server.ssl.truststore.password
bin/kibana-keystore add server.ssl.keystore.password

I think that the current error :
Unable to retrieve version information from Elasticsearch nodes
Is related to the authentication between the elasticsearch cluster and kibana.

Could someone tell me what I need to change in my configuration?
Example of an elasticsearch node

xpack.security.http.ssl:
  enabled: true
  keystore.path: /etc/elasticsearch/new/http.p12
  truststore.path: /etc/elasticsearch/new/http.p12
  client_authentication: required
  verification_mode: full
# Enable encryption and mutual authentication between cluster node
xpack.security.transport.ssl:
  enabled: true
  verification_mode: full
  client_authentication: required
  keystore.path: /etc/elasticsearch/new/node-1.p12
  truststore.path: /etc/elasticsearch/new/node-1.p12

The kibana configuration :


server.ssl.enabled: true
server.ssl.keystore.path: "/etc/kibana/certs/kibana1.p12"
server.ssl.truststore.path: "/etc/kibana/certs/kibana1.p12"
server.ssl.clientAuthentication: required
#server.ssl.certificate: /path/to/your/server.crt
#server.ssl.key: /path/to/your/server.key

# The URLs of the Elasticsearch instances to use for all your queries.
elasticsearch.hosts: ["https://192.168.1.10:9200", "https://192.168.1.20:9200","https://192.168.1.30:9200", "https://192.168.1.40:9200", "https://192.168.1.50:9200"]

elasticsearch.ssl.certificateAuthorities: [ "/etc/kibana/certs/elasticsearch-ca.pem" ]
#elasticsearch.ssl.keystore.path: "/etc/kibana/certs/kibana1.p12"
elasticsearch.ssl.alwaysPresentCertificate: true
# To disregard the validity of SSL certificates, change this setting's value to 'none'.
elasticsearch.ssl.verificationMode: full

Best regards,

You need share Elasticsearch logs to find out the actual errors.

I suspect you don't want this

elasticsearch.ssl.verificationMode: full

unless you have generated your certificate carefully enough to including SANs for each Elasticsearch node. You can try elasticsearch.ssl.verificationMode: certificate instead.

I suspect you don't want this either

server.ssl.clientAuthentication: required

If you plan to access Kibana via just a browser. In general, I suspect you dont' need client authentication for either Elasticsearch or Kibana configurations. You may have some misunderstanding on what it does. Do you have a reason why it is needed?

In your kibana configuration, there is no credentials for Kibana to connect to Elasticsearch. Have you configured that? Either with legacy username/password or a service account token?

Hello @Yang_Wang ,
Thanks for your answers.

For the transport.ssl certificates I generated from this yml file
instances :

>  - name : "node-1"
>     ip :
>       - "192.168.1.10"
>     dns :
>       - "elasticsearch-node1.lab.fr"
>   - nom : "node-2"
>     ip :
>       - "192.168.1.20"
>     dns :
>       - "elasticsearch-node2.lab.fr"
>   - nom : "node-3"
>     ip :
>       - "192.168.1.30"
>     dns :
>       - "elasticsearch-node3.lab.fr"
>   - nom : "node-4"
>     ip :
>       - "192.168.1.40"
>     dns :
>       - "elasticsearch-node4.lab.fr"
>   - nom : "node-5"
>     ip :
>       - "192.168.1.50"
>     dns :
>       - "elasticsearch-node5.lab.fr"
>   - nom : "kibana1"
>     ip :
>       - "192.168.1.80"
>     dns :
>       - "kibana1.lab.fr"
>   - nom : "kibana2"
>     ip :
>       - "192.168.1.90"
>     dns :
>       - "kibana2.lab.fr"
>   - nom : "logstash1"
>     ip :
>       - "192.168.1.60"
>     dns :
>       - "logstash1.lab.fr"
>   - nom : "logstash2"
>     ip :
>       - "192.168.1.70"
>     dns :
>       - "logstash2.lab.fr"

For http.ssl certificates here is an example of the answer I made to the command
./bin/elasticsearch-certutil http

Generate a CSR? [y/N]N

## Do you have an existing Certificate Authority (CA) key-pair that you wish to use to sign your certificate?
Use an existing CA? [y/N]y
## What is the path to your CA?
CA Path: /usr/share/elasticsearch/elastic-stack-ca.p12
Password for elastic-stack-ca.p12:

## How long should your certificates be valid?
For how long should your certificate be valid? [5y]

## Do you wish to generate one certificate per node?
Generate a certificate per node? [y/N]y

## What is the name of node #1?
node #1 name: node-1

## Which hostnames will be used to connect to node-1?

These hostnames will be added as "DNS" names in the "Subject Alternative Name"
(SAN) field in your certificate.

elasticsearch-node1.lab.fr

You entered the following hostnames.

 - elasticsearch-node1.lab.fr

Is this correct [Y/n]y

## Which IP addresses will be used to connect to node-1?

192.168.1.10

You entered the following IP addresses.

 - 192.168.1.10

Is this correct [Y/n]y

## Other certificate options

Key Name: node-1
Subject DN: CN=node-1
Key Size: 2048

Is it correct?

elasticsearch.ssl.verificationMode: full

normally if it is the ip is the hostname (dns) is informed the full mode is not a problem no?

server.ssl.clientAuthentication: required

I commented on it, it doesn't seem necessary.

Here are the error feedbacks from elasticsearch

[2022-10-24T16:12:05,205][WARN ][o.e.h.AbstractHttpServerTransport] [node-4] caught exception while handling client http traffic, closing connection Netty4HttpChannel{localAddress=/192.168.1.40:9200, remoteAddress=/192.168.1.80:35384}
io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors

Caused by: javax.net.ssl.SSLHandshakeException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors

[2022-10-24T16:12:06,290][WARN ][o.e.c.s.DiagnosticTrustManager] [node-4] failed to establish trust with client at [<unknown host>]; the client provided a certificate with subject name [CN=kibana1], fingerprint [c9a35b1c12ec10f39e00c9f920613e4be7dd4bf5], no keyUsage and no extendedKeyUsage; the session uses cipher suite [TLS_AES_256_GCM_SHA384] and protocol [TLSv1.3]; the certificate is issued by [CN=Elastic Certificate Tool Autogenerated CA]; the certificate is signed by (subject [CN=Elastic Certificate Tool Autogenerated CA] fingerprint [a6781c1ac84dbe52b929942e418ff19cc42bf8b7]) which is self-issued; the [CN=Elastic Certificate Tool Autogenerated CA] certificate is not trusted in this ssl context ([xpack.security.http.ssl (with trust configuration: StoreTrustConfig{path=/etc/elasticsearch/new/http.p12, password=<non-empty>, type=PKCS12, algorithm=PKIX})]); this ssl context does trust a certificate with subject [CN=Elastic Certificate Tool Autogenerated CA] but the trusted certificate has fingerprint [2d8da61f994c4a727e8378ae4f76433ed014d6e2]
sun.security.validator.ValidatorException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors

The error message says the server (Elasticsearch) does not trust the client (Kibana)'s certificate. The important bits from the error message are:

the certificate is signed by (subject [CN=Elastic Certificate Tool Autogenerated CA] fingerprint [a6781c1ac84dbe52b929942e418ff19cc42bf8b7])

this ssl context does trust a certificate with subject [CN=Elastic Certificate Tool Autogenerated CA] but the trusted certificate has fingerprint [2d8da61f994c4a727e8378ae4f76433ed014d6e2]

The client cert is signed by a CA of the same name but different fingerprint. I suspect there is a mismatch in the generated files. Is it possible that you have ran the elasticsearch-certutil multiple times and have used different CA file for generating different files (http vs node certs).

You can check whether it is the case by:

  1. Check the issuer signature of Kibana's client cert with keytool -list -v -keystore /etc/kibana/certs/kibana1.p12
  2. Compare with the trusted CA's signature with keytool -list -v -keystore /etc/elasticsearch/new/http.p12

Thank you very much @Yang_Wang

I redid all the certificates carefully.
I have this error now on ES :

[2022-10-25T10:29:54,405][WARN ][o.e.h.AbstractHttpServerTransport] [node-1] caught exception while handling client http traffic, closing connection Netty4HttpChannel{localAddress=/192.168.1.10:9200, remoteAddress=/192.168.1.80:34090}
io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: Empty client certificate chain
Caused by: javax.net.ssl.SSLHandshakeException: Empty client certificate chain

kibana.yml


elasticsearch.ssl.certificateAuthorities: [ "/etc/kibana/certs/elasticsearch-ca.pem" ]
elasticsearch.ssl.verificationMode: full

elasticsearch.yml

xpack.security.http.ssl:
  enabled: true
  keystore.path: /etc/elasticsearch/new/http.p12
  truststore.path: /etc/elasticsearch/new/http.p12
  client_authentication: required
  verification_mode: full

The error means Kibana didn't send any client certificate. You need something like in your Kibana configuration file:

elasticsearch.ssl.keystore.path: "/etc/kibana/certs/kibana1.p12"

You will also need configure the above keystore password with

bin/kibana-keystore add elasticsearch.ssl.keystore.password

Also, have you configured the user (or service token) for Kibana to connect to Elasticsearch? This should be either elasticsearch.username and elasticsearch.password or elasticsearch. serviceAccountToken. You can refer to this page for details.

Thanks @Yang_Wang

I create a service token on es with command :
bin/elasticsearch-service-tokens create elastic/kibana my-token

And I add token in kibana :
elasticsearch.serviceAccountToken: "AAEAAWVsYXN0aWMva2liYW5hL215LXRva2VuOlNUWjNpVUYxUZNXNnFIeWdsSll5XUE"

I change rights in /etc/elasticsearch/service_tokens

chown elasticsearch;elasticsearch etc/elasticsearch/service_tokens
chmod 0644 /etc/elasticsearch/service_tokens

I restart elasticsearch.service on all nodes

I think I have no more errors about "Empty client certificate chain" in ES after add :
elasticsearch.ssl.keystore.path: "/etc/kibana/certs/kibana1.p12"

how to configure the token service for all the nodes of the cluster?

I have this errors now on kibana :

ERROR][elasticsearch-service] Unable to retrieve version information from Elasticsearch nodes. security_exception: [index_not_found_exception] Reason: no such index [.security]
Error: Failure installing common resources shared between all indices. Server is stopping; must stop all async operations
[ERROR][plugins.ruleRegistry] Error: Failure installing common resources shared between all indices. Server is stopping; must stop all async operations

I copied the service_tokens file on all nodes and the connection works!
I want to say thank you for your help!
Thanks to you I will be able to put the secure stack into production.
I am not safe from asking questions when I will have to secure logstash and others.

Hello @Yang_Wang,
I access the kibana login page but when I want to log in username or password is incorrect.
I use this command to fix elastic password :
bin/elasticsearch-reset-password -i -u elastic

In the elastic logs I always get the error when I try to log in Kibana.

[WARN ][o.e.h.AbstractHttpServerTransport] [node-1] caught exception while handling client http traffic, closing connection Netty4HttpChannel{localAddress=/192.168.1.10:9200, remoteAddress=/192.168.1.80:38968}
io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: Empty client certificate chain

yet I added as you told me
elasticsearch.ssl.keystore.path: "/etc/kibana/certs/kibana1.p12"

Why ?

Thank you again for your help

I solved my authentication problem by adding the :
elasticsearch.ssl.alwaysPresentCertificate: true