Upgrade 7.6->7.7 unable to verify the first certificate"

Elastic Stack Kibana
1

I've had my Elastic Stack secured using a public CA certificate for about a year now and upgraded from version to version without issue. Updated Elasticsearch and Kibana from 7.6.2 to 7.7.0 today and everything seemed to work fine until I tried to access the monitoring app. This threw an error:

image

So I go and check out the logs to find this occurred during startup:

{"type":"log","@timestamp":"2020-05-20T17:21:40-05:00","tags":["error","elasticsearch","monitoring"],"pid":4284,"message":"Request error, retrying\nGET https://SERVERFQDN:9200/_xpack => unable to verify the first certificate"}
{"type":"log","@timestamp":"2020-05-20T17:21:40-05:00","tags":["error","elasticsearch","monitoring"],"pid":4284,"message":"Request error, retrying\nGET https://SERVERFQDN:9200/_xpack => unable to verify the first certificate"}

I've tried a number of things and can't seem to figure out what the hell I have configured wrong. I've checked all certificates I'm using and the earliest any of them expire is November of this year. I'm using the same wildcard certificates signed by a public CA in all three applications. Here's my Kibana.yml

server.port: 443
server.host: "ServerFQDN"
server.name: "ServerFQDN"
server.ssl.enabled: true
server.ssl.redirectHttpFromPort: 80
server.ssl.certificate: D:\ElasticStack\Apps\kibana\7.7.0\config\certs\elastic.pem
server.ssl.key: D:\ElasticStack\Apps\kibana\7.7.0\config\certs\elastickey.pem
server.ssl.certificateAuthorities: ["D:/ElasticStack/Apps/kibana/7.7.0/config/certs/inca.pem", "D:/ElasticStack/Apps/kibana/7.7.0/config/certs/ca.pem"]

elasticsearch.hosts: ["https://ServerFQDN:9200"]
elasticsearch.username: "kibana"
elasticsearch.password: ""
elasticsearch.preserveHost: true
elasticsearch.requestTimeout: 120000
elasticsearch.ssl.certificate: D:\ElasticStack\Apps\kibana\7.7.0\config\certs\elastic.pem
elasticsearch.ssl.key: D:\ElasticStack\Apps\kibana\7.7.0\config\certs\elastickey.pem
elasticsearch.ssl.certificateAuthorities: ["D:/ElasticStack/Apps/kibana/7.7.0/config/certs/inca.pem", "D:/ElasticStack/Apps/kibana/7.7.0/config/certs/ca.pem"]

monitoring.ui.elasticsearch.hosts: ["https://ServerFQDN:9200"]
monitoring.ui.elasticsearch.ssl.certificateAuthorities: ["D:/ElasticStack/Apps/kibana/7.7.0/config/certs/inca.pem", "D:/ElasticStack/Apps/kibana/7.7.0/config/certs/ca.pem"]

kibana.defaultAppId: "dashboards"
kibana.autocompleteTimeout: 5000
path.data: D:/ElasticStack/Data/Kibana
pid.file: D:/ElasticStack/Data/Kibana/Kibana.pid

logging.dest: D:/ElasticStack/Logs/Kibana/Kibana.log
logging.silent: false
logging.quiet: false
logging.verbose: true
logging.timezone: America/Chicago
2

bump

3

Surely someone's got something for this...

4

Halp?

5

@wwalker since the settings from monitoring.ui.elasticsearch are the same of elasticsearch.* could we start from removing those?

6

Hello !

I have the same issue as OP.

I always upgraded Kibana without issues , only the stack monitoring doesn't work as usual.

I get the same errors

My settings:

elasticsearch.ssl.certificateAuthorities: [ "/etc/ssl/certs/local_ca.crt" ]
xpack.monitoring.elasticsearch.hosts: ["https://FQDN:9200"]
xpack.monitoring.elasticsearch.ssl.certificateAuthorities: [ "/etc/ssl/certs/local_ca.crt"]
7

Hello,

I also have an issue with the monitoring view after the upgrade to 7.7.
I'm getting back the error:
No Living connections: Check the Elasticsearch Monitoring cluster network connection and refer to the Kibana logs for more information.

Best Regards,
Jo

HTTP 500

8

Yup, if you turn on debugging, you'll see pretty early on in the startup sequence the error message I noted in the original post.

9

@tiagocosta

Is there a solution to this issue ?

Thanks

10

@wwalker @tmartincpp @Jo_De_Troy there is an open bug related with the elasticsearch options on monitoring that we are addressing on https://github.com/elastic/kibana/issues/67668.

For now a suggested workround will be do something like:

monitoring.ui.elasticsearch.ssl.certificateAuthorities: |
  -----BEGIN CERTIFICATE-----
  YOUR CERTIFICATE DATA
  -----END CERTIFICATE-----
monitoring.ui.elasticsearch.ssl.verificationMode: certificate

Please pay attention to the pipe | in the yml as it is important, it preserves line breaks for the PEM certificate.

Cheers

11

Hello,
I've tried that and indeed the error goes away. But now it states there's no monitoring data. The monitoring data is still visible when going to the kibana of the monitoring node. Is that problem specific to me, or do other have the same problem?
Best Regards,
Jo

12

I have this same issue with fresh install of 7.7

13

I added these parameters and it works now.
At first I had "no monitoring data".
For those trying the workaround, don't forget the spaces before the certificate text strings (including ---).

14

Hello,
I got it working after I started using the new parameters instead of the old
monitoring.ui.elasticsearch.* instead of xpack.monitoring.elasticsearch.*

15

Can you post an example of your working 'monitoring' section from kibana.yml please? (obviously snip out the certs :slight_smile: )

16

Here's my working kibana config's monitoring part

monitoring.ui.elasticsearch.hosts: [ "https://{monitoring_cluster_fqdn}:9200" ]
monitoring.ui.elasticsearch.username: xxxxx
monitoring.ui.elasticsearch.password: ******
xpack.monitoring.ui.enabled: true
monitoring.ui.elasticsearch.ssl.certificateAuthorities: |
-----BEGIN CERTIFICATE-----
-----END CERTIFICATE-----

17

Thanks for that. It's good to compare notes :slight_smile:

In the end i removed all monitoring.ui.* and it worked. Suggested by elastic support.
The only thing i had was monitoring.enabled: true no certs nothing.

SSL was enabled by the https to access kibana, i am told.

18

SSL was enabled by the https to access kibana, i am told.

Just to be clear- if you want Kibana to pull the monitoring data from the same place as it is connecting to anyway, then you don't need to specify the monitoring.ui.* settings (as @tiagocosta mentioned up here.
In this case Kibana will use the connection object already established (which works with certificates) rather than create a new one for the monitoring data.

19

As for me I tried the workaround, which worked and like @AndyHunt66 said:
I removed the xpack.monitoring.elasticsearch.* settings I had and it worked without the workaround.

The basic elasticsearch settings were enough since I pull the monitoring data from the same cluster.

20

I tried what was suggested by @tiagocosta, but still continued to get the same errors. I just removed all the monitoring.* settings, set elasticsearch.ssl.verificationMode: certificate, and I'm up and running again.