# Kibana SSL verification Trust Issues
**URL:** https://discuss.elastic.co/t/kibana-ssl-verification-trust-issues/322836
**Category:** Kibana
**Tags:** elastic-stack-security
**Created:** [January 10, 2023, 12:43pm UTC](https://discuss.elastic.co/t/kibana-ssl-verification-trust-issues/322836 "2023-01-10T12:43:17Z")
**Posts on this page:** 20
**Page:** 1
### Author:  [@hanna](https://discuss.elastic.co/u/hanna)
#### Post date: [January 10, 2023, 12:43pm UTC](https://discuss.elastic.co/t/kibana-ssl-verification-trust-issues/322836/1 "2023-01-10T12:43:17Z")
Hi everyone!
I'm trying to set up certificate verification in my elk stack version 8.5.3.
I'm using self generated certificates using my companys ca for each stack component and my cluster works fine as long as I set `elasticsearch.ssl.verificationMode: none` in my kibana configuration.
But whenever I try to verify my ca (either in full or certificate mode) it fails with "kibana server is not ready yet" and my kibana logs hint to a certificate problem:
```auto
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.4.0"},"@timestamp":"2023-01-10T12:42:04.937+01:00","message":"Unable to retrieve version information from Elasticsearch nodes. unable to get issuer certificate","log":{"level":"ERROR","logger":"elasticsearch-service"},"process":{"pid":17138},"trace":{"id":"99a68b8d45a0a85e7624f724d6469b74"},"transaction":{"id":"cee7054ed64c6a14"}}
```
I am using the same ca for the whole stack, why does kibana not trust the elasticsearch certificate? The certificaton works fine with logstash and the elasticsearch nodes.
This is my kibana.yml (the section with certificates):
```auto
...
elasticsearch.hosts: ['https://ELASTIC-01:9200', 'https://ELASTIC-02:9200', 'https://ELASTIC-03:9200']
server.ssl.certificate: /etc/kibana/certs/kibana-cert.pem
server.ssl.key: /etc/kibana/certs/kibana-key.pem
server.ssl.enabled: true
elasticsearch.ssl.certificateAuthorities: ["/etc/kibana/certs/ca.pem"]
elasticsearch.username: "kibana_system"
elasticsearch.password: password
...
```
elasticsearch.ssl.certificateAuthorities: ["/etc/kibana/certs/ca.pem"] is the same ca that signed the elasticsearch http and transport certificates.
The certificate section from elasticsearch.yml:
```auto
xpack.security.enabled: true
xpack.security.http.ssl:
enabled: true
# keystore.path: certs/http.p12
xpack.security.http.ssl.key: certs/elastic.key
xpack.security.http.ssl.certificate: certs/elastic.crt
xpack.security.http.ssl.certificate_authorities: certs/ca.pem
xpack.security.transport.ssl.certificate_authorities: certs/ca.pem
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.client_authentication: required
xpack.security.transport.ssl.key: certs/elastic.key
xpack.security.transport.ssl.certificate: certs/elastic.crt
```
I asked about certificate verification for logstash before in [this post](https://discuss.elastic.co/t/logstash-ssl-verification/322615). Before this, I used the http.p12 certificate store that was created during setup for the automatic kibana enrollment. However enabling logstash ssl certificate verification required me to stop using the keystore and use my own certificates instead. So now I'm facing problems on the other end of the stack...
---
### Author:  [@Ayush\_Mathur](https://discuss.elastic.co/u/Ayush_Mathur)
#### Post date: [January 10, 2023, 1:40pm UTC](https://discuss.elastic.co/t/kibana-ssl-verification-trust-issues/322836/2 "2023-01-10T13:40:29Z")
```auto
server.ssl.certificate: /etc/kibana/certs/kibana-cert.pem
server.ssl.key: /etc/kibana/certs/kibana-key.pem
server.ssl.enabled: true
```
These settings are used for SSL between Kibana server and browser. The issue you are facing is between Kibana and Elasticsearch, so those settings won't have any impact. Instead, you should configure the below:
```auto
elasticsearch.ssl.certificate:
elasticsearch.ssl.key:
elasticsearch.ssl.certificateAuthorities:
elasticsearch.ssl.verificationMode: certificate
```
Also, since Kibana **must** present its certificate to ES to establish connection, the below line must be added as well:
`elasticsearch.ssl.alwaysPresentCertificate: true`
---
### Author:  [@hanna](https://discuss.elastic.co/u/hanna)
#### Post date: [January 10, 2023, 5:01pm UTC](https://discuss.elastic.co/t/kibana-ssl-verification-trust-issues/322836/3 "2023-01-10T17:01:10Z")
I added the settings but I'm still getting the same error:
`Jan 10 17:38:22KIBANA-01 kibana[17263]: [2023-01-10T17:38:22.270+01:00][ERROR][elasticsearch-service] Unable to retrieve version information from Elasticsearch nodes. unable to get issuer certificate`
my kibana.yml looks like this now:
```auto
...
server.ssl.certificate: /etc/kibana/certs/kibana-cert.pem
server.ssl.key: /etc/kibana/certs/kibana-key.pem
server.ssl.enabled: true
elasticsearch.ssl.certificateAuthorities: ["/etc/kibana/certs/ca.pem"]
elasticsearch.username: "kibana_system"
elasticsearch.password: password
elasticsearch.ssl.alwaysPresentCertificate: true
elasticsearch.ssl.certificate: /etc/kibana/certs/kibana-cert.pem
elasticsearch.ssl.key: /etc/kibana/certs/kibana-key.pem
elasticsearch.ssl.verificationMode: certificate
...
```
---
### Author:  [@stephenb](https://discuss.elastic.co/u/stephenb)
#### Post date: [January 10, 2023, 6:36pm UTC](https://discuss.elastic.co/t/kibana-ssl-verification-trust-issues/322836/4 "2023-01-10T18:36:33Z")
Hi @hanna
I am trying to follow along but with all the snippets it is a bit hard...
So it seems you are using Elasticsearch with your generated `pem`s
Per this...
```auto
xpack.security.enabled: true
xpack.security.http.ssl:
enabled: true
# keystore.path: certs/http.p12
xpack.security.http.ssl.key: certs/elastic.key
xpack.security.http.ssl.certificate: certs/elastic.crt
xpack.security.http.ssl.certificate_authorities: certs/ca.pem
xpack.security.transport.ssl.certificate_authorities: certs/ca.pem
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.client_authentication: required
xpack.security.transport.ssl.key: certs/elastic.key
xpack.security.transport.ssl.certificate: certs/elastic.crt
```
And you tried this as your kibana.yml
```auto
elasticsearch.hosts: ['https://ELASTIC-01:9200', 'https://ELASTIC-02:9200', 'https://ELASTIC-03:9200']
elasticsearch.ssl.certificateAuthorities: ["/etc/kibana/certs/ca.pem"]
elasticsearch.username: "kibana_system"
elasticsearch.password: password
```
Where
> [@hanna](#):
>
> elasticsearch.ssl.certificateAuthorities: ["/etc/kibana/certs/ca.pem"] is the same ca that signed the elasticsearch http and transport certificates.
And it does not work... so I am working from there....
so from the kibana machine if you try this
The `-v` verbose and should give us more information
the `-u` you could use the `elastic` or `kibana_system` .. .`elastic` user might be better
`curl -v -u elastic --cacert /etc/kibana/certs/ca.pem https://ELASTIC-01:9200`
I suspect you have a Subject Name missmatch
---
### Author:  [@hanna](https://discuss.elastic.co/u/hanna)
#### Post date: [January 11, 2023, 11:45am UTC](https://discuss.elastic.co/t/kibana-ssl-verification-trust-issues/322836/5 "2023-01-11T11:45:11Z")
Hi @stephenb yes, you understood that correctly.
So I tried the curl command with the kibana\_system as well as the elastic user and got the following output:
```auto
* Trying 172.16.10.55:9200...
* Connected to ELASTIC-01 (172.16.10.55) port 9200 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* CAfile: /etc/kibana/certs/ca.pem
* CApath: /etc/ssl/certs
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS header, Finished (20):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Request CERT (13):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.2 (OUT), TLS header, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.3 (OUT), TLS handshake, Certificate (11):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server did not agree to a protocol
* Server certificate:
* subject: C=DE; ST=Berlin; L=Berlin; O=mycompany; emailAddress=security@mycompany.com; CN=ELASTIC-01
* start date: Jan 11 10:50:36 2023 GMT
* expire date: Feb 12 10:50:36 2024 GMT
* subjectAltName does not match ELASTIC-01
* SSL: no alternative certificate subject name matches target host name 'ELASTIC-01'
* Closing connection 0
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.3 (OUT), TLS alert, close notify (256):
curl: (60) SSL: no alternative certificate subject name matches target host name 'ELASTIC-01'
More details here: https://curl.se/docs/sslcerts.html
curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.
```
Since it says `no alternative certificate subject name matches target host name 'ELASTIC-01'` I tried the request again but with the fqdn "ELASTIC-01.zonelocal", which is in the certficiates subject alternative names. Now
the request is succesful:
```auto
* Trying 172.16.10.55:9200...
* Connected to ELASTIC-01.zonelocal (172.16.10.55) port 9200 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* CAfile: /etc/kibana/certs/ca.pem
* CApath: /etc/ssl/certs
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS header, Finished (20):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Request CERT (13):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.2 (OUT), TLS header, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.3 (OUT), TLS handshake, Certificate (11):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server did not agree to a protocol
* Server certificate:
* subject: C=DE; ST=Berlin; L=Berlin; O=mycompany; emailAddress=security@mycompany.com; CN=ELASTIC-01
* start date: Jan 11 10:50:36 2023 GMT
* expire date: Feb 12 10:50:36 2024 GMT
* subjectAltName: host "ELASTIC-01.zonelocal" matched cert's "ELASTIC-01.zonelocal"
* issuer: C=DE; ST=Berlin; L=Berlin; O=mycompany; emailAddress=security@mycompany.com; CN=companyca
* SSL certificate verify ok.
* Server auth using Basic with user 'kibana_system'
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
> GET / HTTP/1.1
> Host: ELASTIC-01.zonelocal:9200
> Authorization: Basic a2liYW5hX3N5c3RlbTpxd2VydHo=
> User-Agent: curl/7.81.0
> Accept: */*
>
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< X-elastic-product: Elasticsearch
< content-type: application/json
< content-length: 535
<
{
"name" : "ELASTIC-01",
"cluster_name" : "cluster",
"cluster_uuid" : "zZGLy3EfSUqrlw_H3EWORg",
"version" : {
"number" : "8.6.0",
"build_flavor" : "default",
"build_type" : "deb",
"build_hash" : "f67ef2df40237445caa70e2fef79471cc608d70d",
"build_date" : "2023-01-04T09:35:21.782467981Z",
"build_snapshot" : false,
"lucene_version" : "9.4.2",
"minimum_wire_compatibility_version" : "7.17.0",
"minimum_index_compatibility_version" : "7.0.0"
},
"tagline" : "You Know, for Search"
}
* Connection #0 to host ELASTIC-01 left intact
```
So I changed the elasticsearch.hosts in my kibana.yml to:
`elasticsearch.hosts: ['https://ELASTIC-01.zonelocal:9200', 'https://ELASTIC-02.zonelocal:9200', 'https://ELASTIC-03.zonelocal:9200']`
Unfortunately Kibana still doesn't connect to elasticsearch with the same error as always:
`Jan 11 12:31:12 KND-KIBANA-01 kibana[17711]: [2023-01-11T12:31:12.178+01:00][ERROR][elasticsearch-service] Unable to retrieve version information from Elasticsearch nodes. unable to get issuer certificate`
---
### Author:  [@stephenb](https://discuss.elastic.co/u/stephenb)
#### Post date: [January 11, 2023, 2:45pm UTC](https://discuss.elastic.co/t/kibana-ssl-verification-trust-issues/322836/6 "2023-01-11T14:45:04Z")
Curious If you set
`ssl.verification_mode: "none"`
Does Kibana connect?
Comment out the
`elasticsearch.ssl.certificateAuthorities`
---
### Author:  [@Ayush\_Mathur](https://discuss.elastic.co/u/Ayush_Mathur)
#### Post date: [January 11, 2023, 3:18pm UTC](https://discuss.elastic.co/t/kibana-ssl-verification-trust-issues/322836/7 "2023-01-11T15:18:37Z")
Also, can you please try adding `xpack.security.http.ssl.client_authentication: required` to your elasticsearch yaml ?
---
### Author:  [@hanna](https://discuss.elastic.co/u/hanna)
#### Post date: [January 11, 2023, 7:36pm UTC](https://discuss.elastic.co/t/kibana-ssl-verification-trust-issues/322836/8 "2023-01-11T19:36:35Z")
I tried `ssl.verification_mode: "none"` in my kibana.yml which causes my kibana to crash on startup.
Is it possible, that you mean `elasticsearch.ssl.verificationMode: "none"` ? Because if I set it to none, then kibana connects without problems.
---
### Author:  [@hanna](https://discuss.elastic.co/u/hanna)
#### Post date: [January 11, 2023, 7:44pm UTC](https://discuss.elastic.co/t/kibana-ssl-verification-trust-issues/322836/9 "2023-01-11T19:44:01Z")
I just tried this but it didn't help ☹
---
### Author:  [@stephenb](https://discuss.elastic.co/u/stephenb)
#### Post date: [January 11, 2023, 8:14pm UTC](https://discuss.elastic.co/t/kibana-ssl-verification-trust-issues/322836/10 "2023-01-11T20:14:18Z")
> [@hanna](#):
>
> Is it possible, that you mean `elasticsearch.ssl.verificationMode: "none"` ? Because if I set it to none, then kibana connects without problems.
Yes apologies... That is what I meant
So that means that the connectivity and SSL works it is just the cert issue.
And when you ran the curl with the cert from the Kibana server.
It helps if you show the actual command you ran not just the output...
---
### Author:  [@hanna](https://discuss.elastic.co/u/hanna)
#### Post date: [January 12, 2023, 9:38am UTC](https://discuss.elastic.co/t/kibana-ssl-verification-trust-issues/322836/11 "2023-01-12T09:38:52Z")
ok so running
`curl -v -u elastic --cacert /etc/kibana/certs/ca.pem https://ELASTIC-01:9200`
or
`curl -v -u kibana_system --cacert /etc/kibana/certs/ca.pem https://ELASTIC-01:9200`
does not work:
```auto
* Trying 172.16.10.55:9200...
* Connected to ELASTIC-01 (172.16.10.55) port 9200 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* CAfile: /etc/kibana/certs/ca.pem
* CApath: /etc/ssl/certs
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS header, Finished (20):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Request CERT (13):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.2 (OUT), TLS header, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.3 (OUT), TLS handshake, Certificate (11):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server did not agree to a protocol
* Server certificate:
* subject: C=DE; ST=Berlin; L=Berlin; O=mycompany; emailAddress=security@mycompany.com; CN=ELASTIC-01
* start date: Jan 11 10:50:36 2023 GMT
* expire date: Feb 12 10:50:36 2024 GMT
* subjectAltName does not match ELASTIC-01
* SSL: no alternative certificate subject name matches target host name 'ELASTIC-01'
* Closing connection 0
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.3 (OUT), TLS alert, close notify (256):
curl: (60) SSL: no alternative certificate subject name matches target host name 'ELASTIC-01'
More details here: https://curl.se/docs/sslcerts.html
curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.
```
running:
`curl -v -u elastic --cacert /etc/kibana/certs/ca.pem https://ELASTIC-01.zonelocal:9200`
or
`curl -v -u kibana_system --cacert /etc/kibana/certs/ca.pem https://ELASTIC-01.zonelocal:9200`
works:
```auto
* Trying 172.16.10.55:9200...
* Connected to ELASTIC-01.zonelocal (172.16.10.55) port 9200 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* CAfile: /etc/kibana/certs/ca.pem
* CApath: /etc/ssl/certs
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS header, Finished (20):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Request CERT (13):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.2 (OUT), TLS header, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.3 (OUT), TLS handshake, Certificate (11):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server did not agree to a protocol
* Server certificate:
* subject: C=DE; ST=Berlin; L=Berlin; O=mycompany; emailAddress=security@mycompany.com; CN=ELASTIC-01
* start date: Jan 11 10:50:36 2023 GMT
* expire date: Feb 12 10:50:36 2024 GMT
* subjectAltName: host "ELASTIC-01.zonelocal" matched cert's "ELASTIC-01.zonelocal"
* issuer: C=DE; ST=Berlin; L=Berlin; O=mycompany; emailAddress=security@mycompany.com; CN=companyca
* SSL certificate verify ok.
* Server auth using Basic with user 'kibana_system'
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
> GET / HTTP/1.1
> Host: ELASTIC-01.zonelocal:9200
> Authorization: Basic a2liYW5hX3N5c3RlbTpxd2VydHo=
> User-Agent: curl/7.81.0
> Accept: */*
>
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< X-elastic-product: Elasticsearch
< content-type: application/json
< content-length: 535
<
{
"name" : "ELASTIC-01",
"cluster_name" : "cluster",
"cluster_uuid" : "zZGLy3EfSUqrlw_H3EWORg",
"version" : {
"number" : "8.6.0",
"build_flavor" : "default",
"build_type" : "deb",
"build_hash" : "f67ef2df40237445caa70e2fef79471cc608d70d",
"build_date" : "2023-01-04T09:35:21.782467981Z",
"build_snapshot" : false,
"lucene_version" : "9.4.2",
"minimum_wire_compatibility_version" : "7.17.0",
"minimum_index_compatibility_version" : "7.0.0"
},
"tagline" : "You Know, for Search"
}
* Connection #0 to host ELASTIC-01 left intact
```
---
### Author:  [@Rios](https://discuss.elastic.co/u/Rios)
#### Post date: [January 12, 2023, 2:40pm UTC](https://discuss.elastic.co/t/kibana-ssl-verification-trust-issues/322836/12 "2023-01-12T14:40:23Z")
> - subjectAltName does not match ELASTIC-01
> - SSL: no alternative certificate subject name matches target host name 'ELASTIC-01'
Long time ago, I had similar issue. If I remember correctly you should have both ELASTIC-01 and ELASTIC-01.zonelocal in Subject Alternative Names. Can you try to generate a new certificate with both names in altNames?
---
### Author:  [@stephenb](https://discuss.elastic.co/u/stephenb)
#### Post date: [January 12, 2023, 3:07pm UTC](https://discuss.elastic.co/t/kibana-ssl-verification-trust-issues/322836/13 "2023-01-12T15:07:23Z")
Good suggestion @Rios
Also @hanna
I would reduce to only 1 elasticsearch host in the kibana.yml while debugging to make things simpler.
Also if you set the logging higher you might get more information
`logging.level: debug`
---
### Author:  [@hanna](https://discuss.elastic.co/u/hanna)
#### Post date: [January 16, 2023, 12:53pm UTC](https://discuss.elastic.co/t/kibana-ssl-verification-trust-issues/322836/14 "2023-01-16T12:53:02Z")
Ok so I changed my setup to one elasticsearch node and one kibana, to simplify the troubleshooting.
I then created new certificates as suggested by @Rios:
My elasticsearch certificate now has `ELASTIC-01, ELASTIC-01.zonelocal`, and the `ip address` in the subject alternative names.
In my elasticsearch.yml i have set:
```auto
logger.level: debug
```
In kibana.yml:
```auto
logging.loggers:
- name: elasticsearch.query
level: debug
```
(I hope this is similar enough to what you meant @stephenb as "logging.level: debug" didn't work)
Changing the certificates didn't solve my issue, I am getting many log lines like this after kibana loads:
```auto
[2023-01-16T13:35:48.480+01:00][DEBUG][elasticsearch.query.data] [ConnectionError]: unable to get issuer certificate
[2023-01-16T13:35:49.525+01:00][DEBUG][elasticsearch.query.data] [ConnectionError]: unable to get issuer certificate
```
After elasticsearch loads I am getting a lot of these logs:
```auto
[2023-01-16T13:34:19,613][DEBUG][o.a.h.i.c.PoolingHttpClientConnectionManager] [ELASTIC-01] Closing expired connections
[2023-01-16T13:34:29,613][DEBUG][o.a.h.i.c.PoolingHttpClientConnectionManager] [ELASTIC-01] Closing expired connections
```
Please let me know if I should post more logs or what to look for. I didn't find anything certificate related other than the logs I just posted.
Do these setting still work for you to this day @Rios and if so which version of elasticsearch are you using? Because this kind of feels like it might be a bug?
I'm going to try creating my certificates with elasticsearch-certutil and see if theres a problem with the way my certificates are created.
---
### Author:  [@Rios](https://discuss.elastic.co/u/Rios)
#### Post date: [January 16, 2023, 12:58pm UTC](https://discuss.elastic.co/t/kibana-ssl-verification-trust-issues/322836/15 "2023-01-16T12:58:33Z")
Have you tested with curl?
```auto
* subjectAltName: host "ELASTIC-01.zonelocal" matched cert's "ELASTIC-01.zonelocal"
* issuer: C=DE; ST=Berlin; L=Berlin; O=mycompany; emailAddress=security@mycompany.com; CN=companyca
```
This two values are important in a handshake.
---
### Author:  [@hanna](https://discuss.elastic.co/u/hanna)
#### Post date: [January 16, 2023, 1:45pm UTC](https://discuss.elastic.co/t/kibana-ssl-verification-trust-issues/322836/16 "2023-01-16T13:45:29Z")
When I do `curl -v -u kibana_system --cacert /etc/kibana/certs/ca.pem https://ELASTIC-01.zonelocal:9200` this is the output:
```auto
Enter host password for user 'kibana_system':
* Trying 172.16.10.55:9200...
* Connected to ELASTIC-01.zonelocal (172.16.10.55) port 9200 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* CAfile: /etc/kibana/certs/ca.pem
* CApath: /etc/ssl/certs
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS header, Finished (20):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.2 (OUT), TLS header, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server did not agree to a protocol
* Server certificate:
* subject: C=DE; ST=Berlin; L=Berlin; O=mycompany; emailAddress=security@mycompany.com; CN=ELASTIC-01
* start date: Jan 13 09:39:09 2023 GMT
* expire date: Feb 14 09:39:09 2024 GMT
* subjectAltName: host "ELASTIC-01.zonelocal" matched cert's "ELASTIC-01.zonelocal"
* issuer: C=DE; ST=Berlin; L=Berlin; O=mycompany; emailAddress=security@mycompany.com; CN=companyca
* SSL certificate verify ok.
* Server auth using Basic with user 'kibana_system'
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
> GET / HTTP/1.1
> Host: ELASTIC-01.zonelocal:9200
> Authorization: Basic a2liYW5hX3N5c3RlbTpGLUdQZVhDYXRJbG9iTEFKeTlFMw==
> User-Agent: curl/7.81.0
> Accept: */*
>
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< X-elastic-product: Elasticsearch
< content-type: application/json
< content-length: 537
<
{
"name" : "ELASTIC-01",
"cluster_name" : "elasticsearch",
"cluster_uuid" : "BfwE6rBqTpCeWoDu0MC3Sw",
"version" : {
"number" : "8.6.0",
"build_flavor" : "default",
"build_type" : "deb",
"build_hash" : "f67ef2df40237445caa70e2fef79471cc608d70d",
"build_date" : "2023-01-04T09:35:21.782467981Z",
"build_snapshot" : false,
"lucene_version" : "9.4.2",
"minimum_wire_compatibility_version" : "7.17.0",
"minimum_index_compatibility_version" : "7.0.0"
},
"tagline" : "You Know, for Search"
}
```
The names seem to match.
---
### Author:  [@Rios](https://discuss.elastic.co/u/Rios)
#### Post date: [January 16, 2023, 3:36pm UTC](https://discuss.elastic.co/t/kibana-ssl-verification-trust-issues/322836/17 "2023-01-16T15:36:48Z")
Curl is working fine for FQDN.
You have replaced certs, am I right? Have you restarted services?
---
### Author:  [@stephenb](https://discuss.elastic.co/u/stephenb)
#### Post date: [January 16, 2023, 4:41pm UTC](https://discuss.elastic.co/t/kibana-ssl-verification-trust-issues/322836/18 "2023-01-16T16:41:47Z")
Hi @hanna
I think you said you added the IP in the cert did you try that setting in Kibana?
> [@hanna](#):
>
> I'm going to try creating my certificates with elasticsearch-certutil and see if theres a problem with the way my certificates are created.
Yes try that you can use the [instances](https://www.elastic.co/guide/en/elasticsearch/reference/current/certutil.html#certutil-silent) file to set all the names IPs etc.
And apologies the logging setting for Kibana is slightly different.
`logging.root.level: debug`
---
### Author:  [@hanna](https://discuss.elastic.co/u/hanna)
#### Post date: [January 25, 2023, 1:46pm UTC](https://discuss.elastic.co/t/kibana-ssl-verification-trust-issues/322836/19 "2023-01-25T13:46:24Z")
Alright it turned out that the certificates, that I am providing are probably missing some stuff.
A kibana certificate created with elasticsearch-certutil looks like this:
```auto
Bag Attributes
friendlyName: kibana-01
localKeyID: 54 69 6D 65 20 31 36 37 34 31 33 36 31 33 35 31 30 34
subject=CN = KIBANA-01
issuer=CN = Elastic Certificate Tool Autogenerated CA
-----BEGIN CERTIFICATE-----
MIIDY.........redacted........A5q5WgCp
-----END CERTIFICATE-----
Bag Attributes
friendlyName: ca
2.16.840.1.113894.746875.1.1:
subject=CN = Elastic Certificate Tool Autogenerated CA
issuer=CN = Elastic Certificate Tool Autogenerated CA
-----BEGIN CERTIFICATE-----
MIIDS...............redacted..............1aF0=
-----END CERTIFICATE-----
```
The difference to my own certificates is the "Bag Attributes" and that the ca certificate is also appended to the kibana certificate. The kibana-cert.pem that I generated with my own CA ist simply the public certificate:
```auto
-----BEGIN CERTIFICATE-----
MIID......redacted.....WgCp
-----END CERTIFICATE-----
```
I don't have any more time to tinker with it but searching for bag attribute related issues I found [this post](https://discuss.elastic.co/t/is-that-possible-to-use-certificates-signed-by-openssl-instead-of-elasticsearch-certutil/291724), in which someone got it to work. Maybe this will help someone who stumbles across this thread.
Thank you all for your help ☀
---
### Author:  [@v\_melnyk](https://discuss.elastic.co/u/v_melnyk)
#### Post date: [January 25, 2023, 4:38pm UTC](https://discuss.elastic.co/t/kibana-ssl-verification-trust-issues/322836/20 "2023-01-25T16:38:24Z")
Thank you for the feedback, I'm happy to know to that my advice has helped someone 🙂
[Next page](https://discuss.elastic.co/t/kibana-ssl-verification-trust-issues/322836.md?page=2)