Upgrade 7.6->7.7 unable to verify the first certificate"

I've had my Elastic Stack secured using a public CA certificate for about a year now and upgraded from version to version without issue. Updated Elasticsearch and Kibana from 7.6.2 to 7.7.0 today and everything seemed to work fine until I tried to access the monitoring app. This threw an error:

image

So I go and check out the logs to find this occurred during startup:

{"type":"log","@timestamp":"2020-05-20T17:21:40-05:00","tags":["error","elasticsearch","monitoring"],"pid":4284,"message":"Request error, retrying\nGET https://SERVERFQDN:9200/_xpack => unable to verify the first certificate"}
{"type":"log","@timestamp":"2020-05-20T17:21:40-05:00","tags":["error","elasticsearch","monitoring"],"pid":4284,"message":"Request error, retrying\nGET https://SERVERFQDN:9200/_xpack => unable to verify the first certificate"}

I've tried a number of things and can't seem to figure out what the hell I have configured wrong. I've checked all certificates I'm using and the earliest any of them expire is November of this year. I'm using the same wildcard certificates signed by a public CA in all three applications. Here's my Kibana.yml

server.port: 443
server.host: "ServerFQDN"
server.name: "ServerFQDN"
server.ssl.enabled: true
server.ssl.redirectHttpFromPort: 80
server.ssl.certificate: D:\ElasticStack\Apps\kibana\7.7.0\config\certs\elastic.pem
server.ssl.key: D:\ElasticStack\Apps\kibana\7.7.0\config\certs\elastickey.pem
server.ssl.certificateAuthorities: ["D:/ElasticStack/Apps/kibana/7.7.0/config/certs/inca.pem", "D:/ElasticStack/Apps/kibana/7.7.0/config/certs/ca.pem"]

elasticsearch.hosts: ["https://ServerFQDN:9200"]
elasticsearch.username: "kibana"
elasticsearch.password: ""
elasticsearch.preserveHost: true
elasticsearch.requestTimeout: 120000
elasticsearch.ssl.certificate: D:\ElasticStack\Apps\kibana\7.7.0\config\certs\elastic.pem
elasticsearch.ssl.key: D:\ElasticStack\Apps\kibana\7.7.0\config\certs\elastickey.pem
elasticsearch.ssl.certificateAuthorities: ["D:/ElasticStack/Apps/kibana/7.7.0/config/certs/inca.pem", "D:/ElasticStack/Apps/kibana/7.7.0/config/certs/ca.pem"]

monitoring.ui.elasticsearch.hosts: ["https://ServerFQDN:9200"]
monitoring.ui.elasticsearch.ssl.certificateAuthorities: ["D:/ElasticStack/Apps/kibana/7.7.0/config/certs/inca.pem", "D:/ElasticStack/Apps/kibana/7.7.0/config/certs/ca.pem"]

kibana.defaultAppId: "dashboards"
kibana.autocompleteTimeout: 5000
path.data: D:/ElasticStack/Data/Kibana
pid.file: D:/ElasticStack/Data/Kibana/Kibana.pid

logging.dest: D:/ElasticStack/Logs/Kibana/Kibana.log
logging.silent: false
logging.quiet: false
logging.verbose: true
logging.timezone: America/Chicago

bump

Surely someone's got something for this...

Halp?

@wwalker since the settings from monitoring.ui.elasticsearch are the same of elasticsearch.* could we start from removing those?

Hello !

I have the same issue as OP.

I always upgraded Kibana without issues , only the stack monitoring doesn't work as usual.

I get the same errors

My settings:

elasticsearch.ssl.certificateAuthorities: [ "/etc/ssl/certs/local_ca.crt" ]
xpack.monitoring.elasticsearch.hosts: ["https://FQDN:9200"]
xpack.monitoring.elasticsearch.ssl.certificateAuthorities: [ "/etc/ssl/certs/local_ca.crt"]

Hello,

I also have an issue with the monitoring view after the upgrade to 7.7.
I'm getting back the error:
No Living connections: Check the Elasticsearch Monitoring cluster network connection and refer to the Kibana logs for more information.

Best Regards,
Jo

HTTP 500

Yup, if you turn on debugging, you'll see pretty early on in the startup sequence the error message I noted in the original post.

@tiagocosta

Is there a solution to this issue ?

Thanks

@wwalker @tmartincpp @Jo_De_Troy there is an open bug related with the elasticsearch options on monitoring that we are addressing on https://github.com/elastic/kibana/issues/67668.

For now a suggested workround will be do something like:

monitoring.ui.elasticsearch.ssl.certificateAuthorities: |
  -----BEGIN CERTIFICATE-----
  YOUR CERTIFICATE DATA
  -----END CERTIFICATE-----
monitoring.ui.elasticsearch.ssl.verificationMode: certificate 

Please pay attention to the pipe | in the yml as it is important, it preserves line breaks for the PEM certificate.

Cheers

Hello,
I've tried that and indeed the error goes away. But now it states there's no monitoring data. The monitoring data is still visible when going to the kibana of the monitoring node. Is that problem specific to me, or do other have the same problem?
Best Regards,
Jo

I have this same issue with fresh install of 7.7

I added these parameters and it works now.
At first I had "no monitoring data".
For those trying the workaround, don't forget the spaces before the certificate text strings (including ---).